4.11.3node.ipi.for.osp.prod.md

# 在 openshift 4.11 上安装和运行 openstack 

➡️[In English (google translated)](https://github-com.translate.goog/wangzheng422/docker_env/blob/dev/redhat/ocp4/4.11/4.11.3node.ipi.for.osp.prod.md?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp)

本文将讲述，如何在openshift上激活openstack operator，并最终安装一个openstack overcloud。本文的目的，是在home lab上，装出来一个openstack on openshift集群，方便进一步的学习和研究。

由于作者水平，时间和实验环境的限制，本文所使用的方法，会有诸多的不足甚至是错误，欢迎大家指正。

## 背景

在进入正题之前，我们简单的说2句为什么要搞openstack on openshift这个实验。

大家都说，现在是云时代，一般来说，我们指的是使用k8s和容器做为云的底座。但是在k8s出现之前，云是用openstack和虚机作为云的底座的。由于历史原因，openstack在客户群体中，还有很大的装机量，特别是电信行业，大量的私有云，都是用openstack来打造的。如何让这部分群体，想现在的k8s/容器云转型，就是一个很大的问题。

而openstack on openshift，就是推动这种转型的一个尝试/技术方案。我们之前看到的方案，大多数是openstack作为虚机云的底座，然后k8s作为虚机云上的应用/paas部署出来，然后把paas交付给客户。openstack on openshift则是另外一种思路，它把k8s/openshift作为云的底座，然后把openstack作为在容器云上的应用部署出来，然后把这个openstack的overcloud集群交付给客户。

在本文中，我们就展示一下，openshift作为云底座，是如何做到这些的。

## 实验环境和架构图

本次实验，主要是在一台24core, 128G内存的物理机上完成。实验的部署架构图如下。

![](./dia/ocp.osp.drawio.svg)

- 在物理机上，创建了2个bridge（这部分以后会改进，使用ovs, 配合vlan的配置）
- 创建4个kvm，每个kvm有4个网络接口，为了方便，每个kvm都有4块硬盘，但是其实只有worker的那个节点，是需要多出来的3个硬盘的。
- 创建了虚拟BMC，这个是openshift IPI安装过程中需要的。
- 物理机上开启了内存超分，硬盘的thin provision，最大化的榨取物理机的资源。

## 实验步骤

1. 使用IPI模式安装openshift，我们使用 3 node / compact cluster 模式安装
2. 纳管worker node / kvm
3. 安装 cnv, nmstat, sriov, nfs等openshift插件
4. 安装 openstack operator
5. 配置 openstack 参数， 部署 openstack overcloud

## 视频讲解

[<kbd><img src="imgs/2022-11-22-18-22-50.png" width="600"></kbd>](https://www.bilibili.com/video/BV1N44y1X7gP/)

- [bilibili](https://www.bilibili.com/video/BV1N44y1X7gP/)
- [youtube](https://youtu.be/0CQDw_8r5Ac)
- [youtube (in english)](https://youtu.be/L_kSrYkYqlo) , [google translated artical](https://github-com.translate.goog/wangzheng422/docker_env/blob/dev/redhat/ocp4/4.11/4.11.3node.ipi.for.osp.prod.md?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp)

# prepare host

我们的实验是在一台物理机上，资源有限，我们就开启内存超分，让我们可以多多创建虚拟机。另外openstack controller是以虚拟机的形式，运行在openshift node里面的，我们的node已经是kvm了，那么就需要开启嵌套虚拟化的功能。

另外，很重要的一步，就是准备网络，本文创建了2个bridge，这个是openshift官方的方法，虽然并不是非常适用在本次实验，但是也凑合能用，问题是多个vlan混杂在bridge里面了，还有一个问题，就是如果想多个主机部署，vlan下的接口，跨主机是不通的。这个后面再用手动部署的ovs来完善，目前先凑合用。

## memory over commit

我们先看看如何开启内存超分。如果不做如下的操作，多个kvm的内存总和就不能超过物理内存，这个对我们做实验会造成很大的麻烦，我们就开启它。以下操作不用重启系统。

```bash

cat << EOF >> /etc/sysctl.d/99-wzh-sysctl.conf

vm.overcommit_memory = 1

EOF
sysctl --system

```

## nested virtulization

接下来，就是开启嵌套虚拟化。这样，我们就能在kvm里面，再启动一个kvm了。虽然，不得不说，这个里面的kvm是真的慢啊，但是做实验，能做下去最重要，对不。以下操作，不用重启系统。

```bash
# first, go to kvm host to config nested kvm
# https://zhuanlan.zhihu.com/p/35320117
cat /sys/module/kvm_intel/parameters/nested
# 0

cat << EOF > /etc/modprobe.d/kvm-nested.conf
options kvm_intel nested=1  
options kvm-intel enable_shadow_vmcs=1   
options kvm-intel enable_apicv=1         
options kvm-intel ept=1                  
EOF

modprobe -r kvm_intel   #协助掉内核中的kvm_intel模块，注意要在所有虚拟机都关闭的情况下执行
modprobe -a kvm_intel   #重新加载该模块

cat /sys/module/kvm_intel/parameters/nested
# 1

cat /proc/cpuinfo | grep vmx

# on guest os
# if you see this file, means it is success.
ls /dev/kvm

```

## prepare network on 103

接下来，我们就在物理机上创建2个bridge, 分别是baremetal, provisioning，为什么要创建2个bridge，因为这个是openshift IPI安装的需求，但是其实IPI安装是有2个网络模式的，一个是单网络模式，只要baremetal就可以，另外一个模式，才是双网络模式，需要baremetal, provisioning两个网络。那我们能不能只用baremetal这个单网络模式呢？

openstack的官方文档里面，说了必须要provisioning网络，很遗憾，这次官方文档说对了。作者用baremetal单网络模式试过了，在最后部署openstack overcloud computerHCI节点的时候，openstack operator指示openshift给worker node耍操纵系统镜像，这次刷的不是coreos，而是我们提供的rhel镜像。在这一步，可能是openstack operator的限制，镜像必须从provisioning网络提供，也许以后软件升级了，这个双网络模式的要求会取消吧。

这一步操作，对应到架构图，是这部分：

![](dia/ocp.osp.s75.drawio.svg)

### bridge: baremetal

我们先按照openshift官方文档，创建baremetal bridge.

```bash

# 创建实验用虚拟网络

mkdir -p /data/kvm
cd /data/kvm

cat << 'EOF' > /data/kvm/bridge.sh
#!/usr/bin/env bash

PUB_CONN='eno1'
PUB_IP='172.21.6.103/24'
PUB_GW='172.21.6.254'
PUB_DNS='172.21.1.1'

nmcli con down "$PUB_CONN"
nmcli con delete "$PUB_CONN"
nmcli con down baremetal
nmcli con delete baremetal
# RHEL 8.1 appends the word "System" in front of the connection,delete in case it exists
nmcli con down "System $PUB_CONN"
nmcli con delete "System $PUB_CONN"
nmcli connection add ifname baremetal type bridge con-name baremetal ipv4.method 'manual' \
    ipv4.address "$PUB_IP" \
    ipv4.gateway "$PUB_GW" \
    ipv4.dns "$PUB_DNS"
    
nmcli con add type bridge-slave ifname "$PUB_CONN" master baremetal
nmcli con down "$PUB_CONN";pkill dhclient;dhclient baremetal
nmcli con up baremetal
EOF
bash /data/kvm/bridge.sh

nmcli con mod baremetal +ipv4.addresses "192.168.7.103/24"
nmcli con up baremetal

```

### vxlan: provisioning

然后，我们需要创建另外一个bridge, provisioning。如果物理机有2个网卡，当然可以在另外一个网卡上，创建这个bridge，作者之前也是这么做的。但是这里尝试另外一个做法，为以后切换到sdn做一些尝试和准备。

我们就在之前的那个网卡上，创建一个vxlan接口，并且绑定到provisioning bridge上去。

我们照着官方文档做就好了。
- [Chapter 6. Using a VXLAN to create a virtual layer-2 domain for VMs](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/assembly_using-a-vxlan-to-create-a-virtual-layer-2-domain-for-vms_configuring-and-managing-networking)

```bash

nmcli connection add type bridge con-name br-prov ifname br-prov ipv4.method disabled ipv6.method disabled

nmcli con modify br-prov ipv4.method manual ipv4.address 172.22.0.1/24

nmcli connection add type vxlan slave-type bridge con-name br-prov-vxlan5 ifname vxlan5 id 5 local 172.21.6.103 remote 172.21.6.102 master br-prov

nmcli connection up br-prov

bridge fdb show dev vxlan5
# c2:d2:b2:e7:6e:f5 vlan 1 master br-prov permanent
# c2:d2:b2:e7:6e:f5 master br-prov permanent
# 00:00:00:00:00:00 dst 172.21.6.103 self permanent
# ce:1f:f5:9e:f8:7f dst 172.21.6.103 self

cat << EOF > /data/kvm/vxlan5-bridge.xml
<network>
 <name>provisioning</name>
 <forward mode="bridge" />
 <bridge name="br-prov" />
</network>
EOF

virsh net-define /data/kvm/vxlan5-bridge.xml
virsh net-start provisioning
virsh net-autostart provisioning

virsh net-list
#  Name           State    Autostart   Persistent
# -------------------------------------------------
#  default        active   yes         yes
#  provisioning   active   yes         yes

```

## prepare rpm repo on helper

在openshift上部署openstack，目前来说，原理并不复杂，openshift负责提供虚拟机（通过cnv), 和物理机（通过machine api)，这些虚拟机和物理机都已经装好了操纵系统。接下来openshift会准备好一套ansible脚本，管理员到指定pod里面，去运行这个openstack 安装的ansible脚本就好了。后面这一半的操作步骤，和安装openstack是一样的。

既然和安装openstack的过程是一样的，那我们就要按照openstack的要求，准备要rpm repo源。我们还是按照先去海外vps上下载，然后下载回来的做法来搞。

接下来，我们给下载好了rpm repo配置一个，repo config 文件，方便后面openstack安装的时候，导入。

最后，我们做一个自动启动的web server，来给这个rpm repo提供服务。

```bash
# sync repo on vultr
dnf install -y yum-utils

cd mnt/blockstorage/

subscription-manager release --set=8.6

declare -a arr=("rhel-8-for-x86_64-baseos-eus-rpms" 
                "rhel-8-for-x86_64-appstream-eus-rpms" 
                "rhel-8-for-x86_64-highavailability-eus-rpms"
                "ansible-2.9-for-rhel-8-x86_64-rpms"
                openstack-16.2-for-rhel-8-x86_64-rpms
                fast-datapath-for-rhel-8-x86_64-rpms
                )

for i in "${arr[@]}"
do
   dnf reposync --repoid="$i" -m --download-metadata -n --delete
done

# baseos should sync with old versoin
dnf reposync --repoid=rhel-8-for-x86_64-baseos-eus-rpms -m --download-metadata --delete 

# on local / helper
declare -a arr=("rhel-8-for-x86_64-baseos-eus-rpms" 
                "rhel-8-for-x86_64-appstream-eus-rpms" 
                "rhel-8-for-x86_64-highavailability-eus-rpms"
                "ansible-2.9-for-rhel-8-x86_64-rpms"
                openstack-16.2-for-rhel-8-x86_64-rpms
                fast-datapath-for-rhel-8-x86_64-rpms
                )

VAR_IP=158.247.234.245

for i in "${arr[@]}"
do
   rsync -P --delete -arz root@$VAR_IP:/mnt/blockstorage/$i /data/dnf/
done

# after download , we create a repo config file
# this will be used later when install openstack
echo > /data/dnf/osp.repo
for i in "${arr[@]}"
do
cat << EOF >> /data/dnf/osp.repo
[$i]
name=$i
baseurl=http://192.168.7.11:5000/$i
enabled=1
gpgcheck=0
EOF
done

# setup web server startup service
# let the web server auto start
cat << EOF > /etc/systemd/system/local-webserver-osp.service
[Unit]
Description=local-webserver-osp

[Service]
User=root
WorkingDirectory=/data/dnf
ExecStart=/bin/bash -c 'python3 -m http.server 5000'
Restart=always

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload

systemctl enable --now local-webserver-osp.service

```

## lvs config

为了压榨服务器资源，我们还配置lvm thin provision，这样能高效的使用磁盘资源，避免浪费。lvm thin provision简单来说，就是硬盘的超售。

```bash

pvcreate -y /dev/sdb
vgcreate vgdata /dev/sdb

# https://access.redhat.com/articles/766133
lvcreate -y -n poolA -L 500G vgdata
lvcreate -y -n poolA_meta -L 1G vgdata
lvconvert -y --thinpool vgdata/poolA --poolmetadata vgdata/poolA_meta
  # Thin pool volume with chunk size 64.00 KiB can address at most <15.88 TiB of data.
  # WARNING: Converting vgdata/poolA and vgdata/poolA_meta to thin pool's data and metadata volumes with metadata wiping.
  # THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
  # Converted vgdata/poolA and vgdata/poolA_meta to thin pool.

lvextend -l +100%FREE vgdata/poolA
  # Rounding size to boundary between physical extents: <1.09 GiB.
  # Size of logical volume vgdata/poolA_tmeta changed from 1.00 GiB (256 extents) to <1.09 GiB (279 extents).
  # Size of logical volume vgdata/poolA_tdata changed from 500.00 GiB (128000 extents) to <1.09 TiB (285457 extents).
  # Logical volume vgdata/poolA successfully resized.
```

# kvm setup

做完了上面的准备工作，我们就要开始创建kvm了，我们做实验是会反复重装的，所以会首先有清理的脚本。然后我们有另外一些脚本去创建kvm，注意，我们是创建kvm，而不会去启动他们。

## cleanup

我们准备了脚本，来清理kvm，把物理机清理成一个干净的系统。

```bash

create_lv() {
    var_vg=$1
    var_pool=$2
    var_lv=$3
    var_size=$4
    var_action=$5
    lvremove -f $var_vg/$var_lv
    # lvcreate -y -L $var_size -n $var_lv $var_vg
    if [ "$var_action" == "recreate" ]; then
      lvcreate --type thin -n $var_lv -V $var_size --thinpool $var_vg/$var_pool
      wipefs --all --force /dev/$var_vg/$var_lv
    fi
}

virsh destroy ocp4-ipi-osp-master-01
virsh undefine ocp4-ipi-osp-master-01

create_lv vgdata poolA lv-ocp4-ipi-osp-master-01 100G 
create_lv vgdata poolA lv-ocp4-ipi-osp-master-01-data 100G 
create_lv vgdata poolA lv-ocp4-ipi-osp-master-01-data-02 100G 
create_lv vgdata poolA lv-ocp4-ipi-osp-master-01-data-03 100G 

virsh destroy ocp4-ipi-osp-master-02
virsh undefine ocp4-ipi-osp-master-02

create_lv vgdata poolA lv-ocp4-ipi-osp-master-02 100G 
create_lv vgdata poolA lv-ocp4-ipi-osp-master-02-data 100G 
create_lv vgdata poolA lv-ocp4-ipi-osp-master-02-data-02 100G 
create_lv vgdata poolA lv-ocp4-ipi-osp-master-02-data-03 100G 

virsh destroy ocp4-ipi-osp-master-03
virsh undefine ocp4-ipi-osp-master-03

create_lv vgdata poolA lv-ocp4-ipi-osp-master-03 100G 
create_lv vgdata poolA lv-ocp4-ipi-osp-master-03-data 100G 
create_lv vgdata poolA lv-ocp4-ipi-osp-master-03-data-02 100G 
create_lv vgdata poolA lv-ocp4-ipi-osp-master-03-data-03 100G 

virsh destroy ocp4-ipi-osp-worker-01
virsh undefine ocp4-ipi-osp-worker-01

create_lv vgdata poolA lv-ocp4-ipi-osp-worker-01 200G 
create_lv vgdata poolA lv-ocp4-ipi-osp-worker-01-data 100G 
create_lv vgdata poolA lv-ocp4-ipi-osp-worker-01-data-02 100G 
create_lv vgdata poolA lv-ocp4-ipi-osp-worker-01-data-03 100G 

virsh destroy ocp4-ipi-osp-worker-02
virsh undefine ocp4-ipi-osp-worker-02

create_lv vgdata poolA lv-ocp4-ipi-osp-worker-02 200G 
create_lv vgdata poolA lv-ocp4-ipi-osp-worker-02-data 100G 
create_lv vgdata poolA lv-ocp4-ipi-osp-worker-02-data-02 100G 
create_lv vgdata poolA lv-ocp4-ipi-osp-worker-02-data-03 100G 

virsh destroy ocp4-ipi-osp-worker-03
virsh undefine ocp4-ipi-osp-worker-03

create_lv vgdata poolA lv-ocp4-ipi-osp-worker-03 200G 
create_lv vgdata poolA lv-ocp4-ipi-osp-worker-03-data 100G 
create_lv vgdata poolA lv-ocp4-ipi-osp-worker-03-data-02 100G 
create_lv vgdata poolA lv-ocp4-ipi-osp-worker-03-data-03 100G 

VAR_VM=`virsh list --all | grep bootstrap | awk '{print $2}'`
virsh destroy $VAR_VM
virsh undefine $VAR_VM
VAR_POOL=`virsh pool-list --all | grep bootstrap | awk '{print $1}'`
virsh pool-destroy $VAR_POOL
virsh pool-undefine $VAR_POOL
/bin/rm -rf /var/lib/libvirt/openshift-images/*
/bin/rm -rf /var/lib/libvirt/images/*


```

## define kvm on 103

然后，我们就可以开始定义kvm了，这里不能启动kvm，因为定义的kvm没有引导盘，启动了也无法开始安装，IPI模式下，installer会调用virtual bmc redfish接口，给kvm挂载上启动镜像，开始安装过程。

我们为了简单起见，每个kvm都配置了4块硬盘，4个网卡，其实只有worker node这一个kvm会用到4块硬盘。我们的vda硬盘还要大一些，因为要承载集群内的nfs服务器。由于我们配置了lvm thin provision，所以 lv 使用起来就可以肆无忌惮了。

```bash

/bin/rm -rf /var/lib/libvirt/images/*

create_lv() {
    var_vg=$1
    var_pool=$2
    var_lv=$3
    var_size=$4
    var_action=$5
    lvremove -f $var_vg/$var_lv
    # lvcreate -y -L $var_size -n $var_lv $var_vg
    if [ "$var_action" == "recreate" ]; then
      lvcreate --type thin -n $var_lv -V $var_size --thinpool $var_vg/$var_pool
      wipefs --all --force /dev/$var_vg/$var_lv
    fi
}

SNO_MEM=32
export KVM_DIRECTORY=/data/kvm

virsh destroy ocp4-ipi-osp-master-01
virsh undefine ocp4-ipi-osp-master-01

create_lv vgdata poolA lv-ocp4-ipi-osp-master-01 500G recreate
create_lv vgdata poolA lv-ocp4-ipi-osp-master-01-data 100G recreate
create_lv vgdata poolA lv-ocp4-ipi-osp-master-01-data-02 100G recreate
create_lv vgdata poolA lv-ocp4-ipi-osp-master-01-data-03 100G recreate

virt-install --name=ocp4-ipi-osp-master-01 --vcpus=16 --ram=$(($SNO_MEM*1024)) \
  --cpu=host-model \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-master-01,device=disk,bus=virtio,format=raw \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-master-01-data,device=disk,bus=virtio,format=raw \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-master-01-data-02,device=disk,bus=virtio,format=raw \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-master-01-data-03,device=disk,bus=virtio,format=raw \
  --os-variant rhel8.4 \
  --network bridge=baremetal,model=virtio \
  --network network:provisioning,model=virtio \
  --network bridge=baremetal,model=virtio \
  --network bridge=baremetal,model=virtio \
  --print-xml > ${KVM_DIRECTORY}/ocp4-ipi-osp-master-01.xml
virsh define --file ${KVM_DIRECTORY}/ocp4-ipi-osp-master-01.xml

virsh destroy ocp4-ipi-osp-master-02
virsh undefine ocp4-ipi-osp-master-02

create_lv vgdata poolA lv-ocp4-ipi-osp-master-02 500G recreate
create_lv vgdata poolA lv-ocp4-ipi-osp-master-02-data 100G recreate
create_lv vgdata poolA lv-ocp4-ipi-osp-master-02-data-02 100G recreate
create_lv vgdata poolA lv-ocp4-ipi-osp-master-02-data-03 100G recreate

virt-install --name=ocp4-ipi-osp-master-02 --vcpus=16 --ram=$(($SNO_MEM*1024)) \
  --cpu=host-model \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-master-02,device=disk,bus=virtio,format=raw \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-master-02-data,device=disk,bus=virtio,format=raw \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-master-02-data-02,device=disk,bus=virtio,format=raw \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-master-02-data-03,device=disk,bus=virtio,format=raw \
  --os-variant rhel8.4 \
  --network bridge=baremetal,model=virtio \
  --network network:provisioning,model=virtio \
  --network bridge=baremetal,model=virtio \
  --network bridge=baremetal,model=virtio \
  --print-xml > ${KVM_DIRECTORY}/ocp4-ipi-osp-master-02.xml
virsh define --file ${KVM_DIRECTORY}/ocp4-ipi-osp-master-02.xml


# SNO_MEM=64

virsh destroy ocp4-ipi-osp-master-03
virsh undefine ocp4-ipi-osp-master-03

create_lv vgdata poolA lv-ocp4-ipi-osp-master-03 500G recreate
create_lv vgdata poolA lv-ocp4-ipi-osp-master-03-data 100G recreate
create_lv vgdata poolA lv-ocp4-ipi-osp-master-03-data-02 100G recreate
create_lv vgdata poolA lv-ocp4-ipi-osp-master-03-data-03 100G recreate

virt-install --name=ocp4-ipi-osp-master-03 --vcpus=16 --ram=$(($SNO_MEM*1024)) \
  --cpu=host-model \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-master-03,device=disk,bus=virtio,format=raw \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-master-03-data,device=disk,bus=virtio,format=raw \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-master-03-data-02,device=disk,bus=virtio,format=raw \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-master-03-data-03,device=disk,bus=virtio,format=raw \
  --os-variant rhel8.4 \
  --network bridge=baremetal,model=virtio \
  --network network:provisioning,model=virtio \
  --network bridge=baremetal,model=virtio \
  --network bridge=baremetal,model=virtio \
  --print-xml > ${KVM_DIRECTORY}/ocp4-ipi-osp-master-03.xml
virsh define --file ${KVM_DIRECTORY}/ocp4-ipi-osp-master-03.xml

SNO_MEM=16

virsh destroy ocp4-ipi-osp-worker-01
virsh undefine ocp4-ipi-osp-worker-01

create_lv vgdata poolA lv-ocp4-ipi-osp-worker-01 500G recreate
create_lv vgdata poolA lv-ocp4-ipi-osp-worker-01-data 100G recreate
create_lv vgdata poolA lv-ocp4-ipi-osp-worker-01-data-02 100G recreate
create_lv vgdata poolA lv-ocp4-ipi-osp-worker-01-data-03 100G recreate

virt-install --name=ocp4-ipi-osp-worker-01 --vcpus=16 --ram=$(($SNO_MEM*1024)) \
  --cpu=host-model \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-worker-01,device=disk,bus=virtio,format=raw \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-worker-01-data,device=disk,bus=virtio,format=raw \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-worker-01-data-02,device=disk,bus=virtio,format=raw \
  --disk path=/dev/vgdata/lv-ocp4-ipi-osp-worker-01-data-03,device=disk,bus=virtio,format=raw \
  --os-variant rhel8.4 \
  --network bridge=baremetal,model=virtio \
  --network network:provisioning,model=virtio \
  --network bridge=baremetal,model=virtio \
  --network bridge=baremetal,model=virtio \
  --print-xml > ${KVM_DIRECTORY}/ocp4-ipi-osp-worker-01.xml
virsh define --file ${KVM_DIRECTORY}/ocp4-ipi-osp-worker-01.xml

```

## bmc simulator

定义了kvm，我们需要配套的virtual BMC / redfish 接口来控制他们，这都是为了模拟真实的物理机，在真实的物理机场景下，openshift installer会调用redfish接口来控制物理机。

我们选用openstack项目的sushy工具来做这个virtual BMC。运行一个sushy实例，就可以管理同一个物理机上的所有kvm实例，简单易用。

最后，我们使用systemd来定义一个自动启动的服务，来运行sushy.

这一步操作，对应到架构图，是这部分：

![](dia/ocp.osp.s77.drawio.svg)

```bash
# try to install and run it manually
dnf -y install python3-pip
pip3 install --user sushy-tools

mkdir -p /etc/crts
scp root@192.168.7.11:/etc/crts/* /etc/crts/

/root/.local/bin/sushy-emulator -i 0.0.0.0 --ssl-certificate /etc/crts/redhat.ren.crt --ssl-key /etc/crts/redhat.ren.key

# try to deploy as systemd service
cat << EOF > /etc/systemd/system/sushy-emulator.service
[Unit]
Description=sushy-emulator

[Service]
User=root
WorkingDirectory=/root
ExecStart=/bin/bash -c '/root/.local/bin/sushy-emulator -i 0.0.0.0 --ssl-certificate /etc/crts/redhat.ren.crt --ssl-key /etc/crts/redhat.ren.key'
Restart=always

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload

systemctl enable --now sushy-emulator.service

```

## get mac and vm list on 103

有了virtual BMC，我们就要抽取一些openshift installer需要用到的参数，一个是kvm的mac地址，一个是redfish里面需要的uuid。

我们使用如下的脚本，来自动的得到，并且上传到 helper 节点去。

```bash

# on helper clean all
/bin/rm -f /data/install/mac.list.*
/bin/rm -f /data/install/vm.list.*

# back to 103
cd /data/kvm/
for i in ocp4-ipi-osp-master-0{1..3} ocp4-ipi-osp-worker-0{1..1}
do
  echo -ne "${i}\t" ; 
  virsh dumpxml ${i} | grep "mac address" | cut -d\' -f2 | tr '\n' '\t'
  echo 
done > mac.list.103
cat /data/kvm/mac.list.103
# ocp4-ipi-osp-master-01  52:54:00:67:64:5f       52:54:00:e8:28:e7       52:54:00:4a:a4:39
# ocp4-ipi-osp-master-02  52:54:00:ac:ed:36       52:54:00:b5:34:c4       52:54:00:87:36:75
# ocp4-ipi-osp-master-03  52:54:00:ae:72:e5       52:54:00:87:19:c2       52:54:00:99:55:12
# ocp4-ipi-osp-worker-01  52:54:00:17:b2:2d       52:54:00:ca:74:c0       52:54:00:f4:5e:a8

cat << 'EOF' > redfish.sh
#!/usr/bin/env bash

curl -k -s https://127.0.0.1:8000/redfish/v1/Systems/ | jq -r '.Members[]."@odata.id"' >  list

while read -r line; do
    curl -k -s https://127.0.0.1:8000/$line | jq -j '.Id, " ", .Name, "\n" '
done < list

EOF
bash redfish.sh | grep ipi > /data/kvm/vm.list.103
cat /data/kvm/vm.list.103
# 6b9a4f6b-d751-4fd5-9493-39792039e9e2 ocp4-ipi-osp-worker-01
# 1a2d1e2a-5f50-49cf-920e-11f7b7f136dc ocp4-ipi-osp-master-02
# 9c7085a2-ed0c-4cbf-94ca-065d3e8db335 ocp4-ipi-osp-master-01
# 14474c89-152c-4580-8bbb-7f03e4e370e0 ocp4-ipi-osp-master-03

scp /data/kvm/{mac,vm}.list.* root@192.168.7.11:/data/install/

```

# on helper node

终于所有的准备工作都做完了，我们开始在helper上面进行openshift的安装。在这之前，还有一个配置helper节点的步骤，主要是配置dns服务之类的，在这里就不重复了，有需要了解的，可以看[这里的文档](./4.11.helper.node.oc.mirror.md)

## get installer binary

我们先要从安装文件目录中，得到installer的二进制文件。

```bash

# switch to you install version

export BUILDNUMBER=4.11.6

pushd /data/ocp4/${BUILDNUMBER}
tar -xzf openshift-client-linux-${BUILDNUMBER}.tar.gz -C /usr/local/bin/
tar -xzf openshift-install-linux-${BUILDNUMBER}.tar.gz -C /usr/local/bin/
tar -xzf oc-mirror.tar.gz -C /usr/local/bin/
chmod +x /usr/local/bin/oc-mirror
install -m 755 /data/ocp4/clients/butane-amd64 /usr/local/bin/butane
install -m 755 /data/ocp4/clients/coreos-installer_amd64 /usr/local/bin/coreos-installer
popd

```

## prepare web server for iso/images

接下来，我们准备一个自动启动的 web server，提供一些iso等镜像的下载服务。

```bash
############################
# as root create web server
cd /data/ocp4

python3 -m http.server 8080

cat << EOF > /etc/systemd/system/local-webserver.service
[Unit]
Description=local-webserver

[Service]
User=root
WorkingDirectory=/data/ocp4
ExecStart=/bin/bash -c 'python3 -m http.server 8080'
Restart=always

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload

systemctl enable --now local-webserver.service

# end as root
############################
```

## create the install yaml

接下来我们创建安装配置文件。这里面最关键的就是那个yaml模板，我们在模板里面，启动IPI安装模式，并且配置3个master的redfish接口信息，并启用静态IP安装的方法，配置了静态IP信息。

安装配置yaml文件创建后，我们调用installer，把他们转化成ignition等真正的安装配置文件，并且和baremetal installer二进制文件一起，传递到物理机上。

这里面有2个二进制文件，一个是openshift installer，这个一般场景下，比如对接公有云，私有云，就够了，它会创建ignition文件，并且调用各种云的接口，创建虚拟机，开始安装。

但是如果是baremetal场景，有一个单独的baremetal installer二进制文件，它读取配置文件，调用物理机BMC接口信息，来开始安装，这个区别是目前openshift版本上的情况，不知道未来会不会有变化。

```bash
# create a user and create the cluster under the user

useradd -m 3nodeipi

su - 3nodeipi

ssh-keygen

cat << EOF > ~/.ssh/config
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null
EOF

chmod 600 ~/.ssh/config

cat << 'EOF' >> ~/.bashrc

export BASE_DIR='/home/3nodeipi/'

EOF

export BASE_DIR='/home/3nodeipi/'
export BUILDNUMBER=4.11.6

mkdir -p ${BASE_DIR}/data/{sno/disconnected,install}

# set some parameter of you rcluster

NODE_SSH_KEY="$(cat ${BASE_DIR}/.ssh/id_rsa.pub)"
INSTALL_IMAGE_REGISTRY=quaylab.infra.redhat.ren:8443

PULL_SECRET='{"auths":{"registry.redhat.io": {"auth": "ZHVtbXk6ZHVtbXk=","email": "noemail@localhost"},"registry.ocp4.redhat.ren:5443": {"auth": "ZHVtbXk6ZHVtbXk=","email": "noemail@localhost"},"'${INSTALL_IMAGE_REGISTRY}'": {"auth": "'$( echo -n 'admin:shadowman' | openssl base64 )'","email": "noemail@localhost"}}}'

NTP_SERVER=192.168.7.11
HELP_SERVER=192.168.7.11
KVM_HOST=192.168.7.11
API_VIP=192.168.7.100
INGRESS_VIP=192.168.7.101
CLUSTER_PROVISION_IP=192.168.7.103
BOOTSTRAP_IP=192.168.7.12

# 定义单节点集群的节点信息
SNO_CLUSTER_NAME=acm-demo-one
SNO_BASE_DOMAIN=redhat.ren

BOOTSTRAP_IP=192.168.7.22
MASTER_01_IP=192.168.7.23
MASTER_02_IP=192.168.7.24
MASTER_03_IP=192.168.7.25
WORKER_01_IP=192.168.7.26

BOOTSTRAP_HOSTNAME=bootstrap-demo
MASTER_01_HOSTNAME=master-01-demo
MASTER_02_HOSTNAME=master-02-demo
MASTER_03_HOSTNAME=master-03-demo
WORKER_01_HOSTNAME=worker-01-demo

BOOTSTRAP_INTERFACE=enp1s0
MASTER_01_INTERFACE=enp1s0
MASTER_02_INTERFACE=enp1s0
MASTER_03_INTERFACE=enp1s0
WORKER_01_INTERFACE=enp1s0

BOOTSTRAP_DISK=/dev/vda
MASTER_01_DISK=/dev/vda
MASTER_02_DISK=/dev/vda
MASTER_03_DISK=/dev/vda
WORKER_01_DISK=/dev/vda

OCP_GW=192.168.7.11
OCP_NETMASK=255.255.255.0
OCP_NETMASK_S=24
OCP_DNS=192.168.7.11

# echo ${SNO_IF_MAC} > /data/sno/sno.mac

mkdir -p ${BASE_DIR}/data/install
cd ${BASE_DIR}/data/install

/bin/rm -rf *.ign .openshift_install_state.json auth bootstrap manifests master*[0-9] worker*[0-9] openshift

cat << EOF > ${BASE_DIR}/data/install/install-config.yaml 
apiVersion: v1
baseDomain: $SNO_BASE_DOMAIN
compute:
- name: worker
  replicas: 0
controlPlane:
  name: master
  replicas: 3 
metadata:
  name: $SNO_CLUSTER_NAME
networking:
  # OVNKubernetes , OpenShiftSDN
  networkType: OVNKubernetes
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - 172.31.0.0/16
  machineNetwork:
  - cidr: 192.168.7.0/24
pullSecret: '${PULL_SECRET}'
sshKey: |
$( cat ${BASE_DIR}/.ssh/id_rsa.pub | sed 's/^/   /g' )
additionalTrustBundle: |
$( cat /etc/crts/redhat.ren.ca.crt | sed 's/^/   /g' )
imageContentSources:
- mirrors:
  - ${INSTALL_IMAGE_REGISTRY}/openshift/release-images
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - ${INSTALL_IMAGE_REGISTRY}/openshift/release
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
platform:
  baremetal:
    apiVIP: $API_VIP
    ingressVIP: $INGRESS_VIP
    provisioningNetwork: "Managed"
    provisioningNetworkCIDR: 172.22.0.0/24
    provisioningNetworkInterface: enp2s0
    provisioningBridge: br-prov
    clusterProvisioningIP: 172.22.0.6
    bootstrapProvisioningIP: 172.22.0.7
    bootstrapExternalStaticIP: 192.168.7.22/24
    bootstrapExternalStaticGateway: 192.168.7.11
    externalBridge: baremetal
    bootstrapOSImage: http://192.168.7.11:8080/rhcos-qemu.x86_64.qcow2.gz?sha256=$(zcat /data/ocp4/rhcos-qemu.x86_64.qcow2.gz | sha256sum | awk '{print $1}')
    clusterOSImage: http://192.168.7.11:8080/rhcos-openstack.x86_64.qcow2.gz?sha256=$(sha256sum /data/ocp4/rhcos-openstack.x86_64.qcow2.gz | awk '{print $1}')
    hosts:
      - name: ocp4-ipi-osp-master-01
        role: master
        bootMode: legacy
        bmc:
          address: redfish-virtualmedia://192.168.7.103:8000/redfish/v1/Systems/$(cat /data/install/vm.list.* | grep master-01 | awk '{print $1}')
          username: admin
          password: password
          disableCertificateVerification: True
        bootMACAddress: $(cat /data/install/mac.list.* | grep master-01 | awk '{print $2}')
        rootDeviceHints:
          deviceName: "$MASTER_01_DISK"
        networkConfig: 
          dns-resolver:
            config:
              server:
              - ${OCP_DNS}
          interfaces:
          - ipv4:
              address:
              - ip: ${MASTER_01_IP}
                prefix-length: ${OCP_NETMASK_S}
              dhcp: false
              enabled: true
            name: ${MASTER_01_INTERFACE}
            state: up
            type: ethernet
          routes:
            config:
            - destination: 0.0.0.0/0
              next-hop-address: ${OCP_GW}
              next-hop-interface: ${MASTER_01_INTERFACE}
              table-id: 254
      - name: ocp4-ipi-osp-master-02
        role: master
        bootMode: legacy
        bmc:
          address: redfish-virtualmedia://192.168.7.103:8000/redfish/v1/Systems/$(cat /data/install/vm.list.* | grep master-02 | awk '{print $1}')
          username: admin
          password: password
          disableCertificateVerification: True
        bootMACAddress: $(cat /data/install/mac.list.* | grep master-02 | awk '{print $2}')
        rootDeviceHints:
          deviceName: "$MASTER_02_DISK"
        networkConfig: 
          dns-resolver:
            config:
              server:
              - ${OCP_DNS}
          interfaces:
          - ipv4:
              address:
              - ip: ${MASTER_02_IP}
                prefix-length: ${OCP_NETMASK_S}
              dhcp: false
              enabled: true
            name: ${MASTER_02_INTERFACE}
            state: up
            type: ethernet
          routes:
            config:
            - destination: 0.0.0.0/0
              next-hop-address: ${OCP_GW}
              next-hop-interface: ${MASTER_02_INTERFACE}
              table-id: 254
      - name: ocp4-ipi-osp-master-03
        role: master
        bootMode: legacy
        bmc:
          address: redfish-virtualmedia://192.168.7.103:8000/redfish/v1/Systems/$(cat /data/install/vm.list.* | grep master-03 | awk '{print $1}')
          username: admin
          password: password
          disableCertificateVerification: True
        bootMACAddress: $(cat /data/install/mac.list.* | grep master-03 | awk '{print $2}')
        rootDeviceHints:
          deviceName: "$MASTER_03_DISK"
        networkConfig: 
          dns-resolver:
            config:
              server:
              - ${OCP_DNS}
          interfaces:
          - ipv4:
              address:
              - ip: ${MASTER_03_IP}
                prefix-length: ${OCP_NETMASK_S}
              dhcp: false
              enabled: true
            name: ${MASTER_03_INTERFACE}
            state: up
            type: ethernet
          routes:
            config:
            - destination: 0.0.0.0/0
              next-hop-address: ${OCP_GW}
              next-hop-interface: ${MASTER_03_INTERFACE}
              table-id: 254
EOF

/bin/cp -f ${BASE_DIR}/data/install/install-config.yaml ${BASE_DIR}/data/install/install-config.yaml.bak

/data/ocp4/${BUILDNUMBER}/openshift-baremetal-install --dir ${BASE_DIR}/data/install/ create manifests

/bin/cp -f  /data/ocp4/ansible-helper/files/* ${BASE_DIR}/data/install/openshift/

#############################################
# run as root if you have not run below, at least one time
# it will generate registry configuration
# copy image registry proxy related config
cd /data/ocp4
bash image.registries.conf.sh nexus.infra.redhat.ren:8083

/bin/cp -f /data/ocp4/image.registries.conf /etc/containers/registries.conf.d/
#############################################

/bin/cp -f /data/ocp4/99-worker-container-registries.yaml ${BASE_DIR}/data/install/openshift
/bin/cp -f /data/ocp4/99-master-container-registries.yaml ${BASE_DIR}/data/install/openshift

cd ${BASE_DIR}/data/install/


# then, we copy baremetal install binary to kvm host

sshpass -p panpan ssh-copy-id root@172.21.6.103

scp /data/ocp4/${BUILDNUMBER}/openshift-baremetal-install root@172.21.6.103:/usr/local/bin/

# the, we copy configuration files to kvm host

cat << EOF > ${BASE_DIR}/data/install/scp.sh
ssh root@172.21.6.103 "rm -rf /data/install;"

scp -r ${BASE_DIR}/data/install root@172.21.6.103:/data/install
EOF

bash ${BASE_DIR}/data/install/scp.sh

```

# kvm host (103) to begin install

到现在位置，万事俱备了，我们就可以在物理机上真正的开始安装了。到这一步，我们没有特别需要做的，因为是IPI模式，全自动，我们运行命令，等着安装成功的结果，并且把各种密码输出记录下来就好了。

```bash

cd /data/install
openshift-baremetal-install --dir /data/install/ --log-level debug create cluster
# ......
# INFO Install complete!
# INFO To access the cluster as the system:admin user when using 'oc', run
# INFO     export KUBECONFIG=/data/install/auth/kubeconfig
# INFO Access the OpenShift web-console here: https://console-openshift-console.apps.acm-demo-one.redhat.ren
# INFO Login to the console with user: "kubeadmin", and password: "JgTXJ-d9Nsb-QHGS2-Puor3"
# DEBUG Time elapsed per stage:
# DEBUG          bootstrap: 23s
# DEBUG            masters: 16m31s
# DEBUG Bootstrap Complete: 19m11s
# DEBUG  Bootstrap Destroy: 11s
# DEBUG  Cluster Operators: 7m10s
# INFO Time elapsed: 43m37s

# tail -f /data/install/.openshift_install.log

```

## on helper to see result

我们需要把物理机上的密钥文件等信息，传回helper节点。方便我们后续的操作。

```bash
# on helper node
scp -r root@172.21.6.103:/data/install/auth ${BASE_DIR}/data/install/auth

cd ${BASE_DIR}/data/install
export KUBECONFIG=${BASE_DIR}/data/install/auth/kubeconfig
echo "export KUBECONFIG=${BASE_DIR}/data/install/auth/kubeconfig" >> ~/.bashrc
# oc completion bash | sudo tee /etc/bash_completion.d/openshift > /dev/null


# if you power off cluster for long time
# you will need to re-approve the csr
oc get csr | grep -v Approved
oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve

```

## password login and oc config

安装完成了，我们要配置一些节点ssh登录的配置。openshift默认的ssh登录，是禁用root登录的，并且启动了time out 机制，这个让我们做实验非常难受和不便，我们在这里就使用脚本，打开这些限制。最终，我们可以轻松的远程root直接用密码登录了。

```bash

# init setting for helper node
cat << EOF > ~/.ssh/config
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null
EOF
chmod 600 ~/.ssh/config


cat > ${BASE_DIR}/data/install/crack.txt << EOF

echo redhat | sudo passwd --stdin root

sudo sed -i "s|^PasswordAuthentication no$|PasswordAuthentication yes|g" /etc/ssh/sshd_config
sudo sed -i "s|^PermitRootLogin no$|PermitRootLogin yes|g" /etc/ssh/sshd_config
sudo sed -i "s|^#ClientAliveInterval 180$|ClientAliveInterval 1800|g" /etc/ssh/sshd_config

sudo systemctl restart sshd

sudo sh -c 'echo "export KUBECONFIG=/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/localhost.kubeconfig" >> /root/.bashrc'

sudo sh -c 'echo "RET=\\\`oc config use-context system:admin\\\`" >> /root/.bashrc'

EOF

for i in 23 24 25
do
  ssh core@192.168.7.$i < ${BASE_DIR}/data/install/crack.txt
done

```

## from other host

能远程密码登录了，我们还希望能自动ssh登录，由于openshift节点很多，一台一台的去配置比较麻烦，我们这里提供了脚本，可以批量的来搞。

```bash
# https://unix.stackexchange.com/questions/230084/send-the-password-through-stdin-in-ssh-copy-id
dnf install -y sshpass

for i in 23 24 25
do
  sshpass -p 'redhat' ssh-copy-id root@192.168.7.$i
done

```

## power off

home lab的特点，是为了节电，没人用的时候，需要关机，那么我们就提供这样的脚本，来方便openshift集群关机操作。

```bash

for i in 23 24 25
do
  ssh root@192.168.7.$i poweroff
done

```

## reboot

有的时候，openshift节点需要全部来一次reboot，来排除错误。这里也有脚本来帮忙。

```bash

for i in 23 24 25
do
  ssh root@192.168.7.$i reboot
done

```

## power on

同样，我们有脚本帮助批量启动虚拟机。

```bash

# or

for i in {1..3}
do
  virsh start ocp4-ipi-osp-master-0$i
done

```

## check info

我们日常还会有一些集群各个节点收集信息，脚本操作的工作，也提供脚本模板，帮助日常工作。

```bash

for i in 23 24 25
do
  ssh root@192.168.7.$i "ip a"
done

cat > ${BASE_DIR}/data/install/crack.txt << 'EOF'

for i in {3..8}
do
  nmcli con down enp${i}s0
  nmcli con del enp${i}s0
done

EOF

for i in 23 24 25
do
  ssh root@192.168.7.$i < ${BASE_DIR}/data/install/crack.txt
done

```

## try to deploy gitea

接下来，需要在helper节点上，部署一个git服务，这个是因为openstack在安装过程中，会先把安装脚本和配置，上传到git服务器上，作为一个git commit，然后真正的部署动作，会从这个git commit上下载，执行。

我们用gitea来提供这个git服务，安装过程网上教程一堆，我们就用最简单的方法来装。但是openstack对git服务有特殊的要求，就是git服务必须使用ssh通道来提供服务，这个就需要测试了，而我们使用了非标准的ssh端口来提供这个服务，也导致了后面一连串的不兼容错误。

不管怎么说，通过ssh访问git服务，肯定要配置出来，并且，我们还要配置ssh key 认证，用密钥的方式的访问。

![](imgs/2022-11-18-23-19-19.png)

最后，给出了测试ssh访问git服务的命令行，方便验证。

```bash
rm -rf /data/ccn/gitea
mkdir -p /data/ccn/gitea
chown -R 1000:1000 /data/ccn/gitea

podman run -d --replace --name gitea \
  -v /data/ccn/gitea:/data:Z \
  -v /etc/localtime:/etc/localtime:ro \
  -e USER_UID=1000 \
  -e USER_GID=1000 \
  -p 10090:3000 \
  -p 10022:22 \
  docker.io/gitea/gitea:1.17.3

# use systemd to auto-run gitea
podman generate systemd --files --name gitea
/bin/cp -Zf container-gitea.service  /etc/systemd/system/

systemctl daemon-reload
systemctl enable --now  container-gitea.service

# http://quaylab.infra.redhat.ren:10090/
# root / redhat

# setup ssh key for gitea

# test the ssh git access
ssh -T -p 10022 git@quaylab.infra.redhat.ren

git clone ssh://git@quaylab.infra.redhat.ren:10022/root/demo.git

```

## add baremetal host

IPI 模式下，添加一个新节点非常方便，只要定义一个BareMetalHost就好了。我们做osp on ocp的实验，添加一个worker node就可以了，后面osp operator会重新格式化这个worker node，然后把他加到osp cluster里面去。

配置是很简单，但是后面具体openshift是做了什么，让它能管理这个baremetal节点呢？ 经过反复的实验，作者大概归纳了相关的行为如下：
1. 当第一次定义BareMetalHost的以后，ocp会调用redfish端口，启动这个节点，同时挂载一个rhcos 的iso，启动这个节点，这个iso是定制过的，会有一些网络参数，还有定义了自启动的服务。启动了以后，会默认启动一个ironic agent，这个ironic agent会连接 machine api，去下载任务，没发现什么特殊的任务的时候，它会探测一下主机环境，比如有多少core, memory, 存储等等，上报machine api以后，就自动关机了。
2. 如果BareMetalHost里面还定义了image，那么ironic agent会下载这个镜像（之前在metal3 service里面转化过了），然后把它写到硬盘上，然后重启。这部分自动运行的命令，作者给记录下来了，[在这里](../../notes/2022/2022.11.clear.partition.md)

这一步操作，对应到架构图，是这部分：

![](dia/ocp.osp.s80.drawio.svg)

```bash

cd ${BASE_DIR}/data/install/

cat << EOF > ${BASE_DIR}/data/install/bmh-01.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: worker-1-bmc-secret
type: Opaque
data:
  username: $(echo -ne "admin" | base64)
  password: $(echo -ne "password" | base64)
---
apiVersion: v1
kind: Secret
metadata:
  name: ocp4-ipi-osp-worker-01-network-config-secret
type: Opaque
stringData:
  nmstate: |
    dns-resolver:
      config:
        server:
        - 192.168.7.11
    interfaces:
    - ipv4:
        address:
        - ip: 192.168.7.26
          prefix-length: 24
        dhcp: false
        enabled: true
      name: enp1s0
      state: up
      type: ethernet
    routes:
      config:
      - destination: 0.0.0.0/0
        next-hop-address: 192.168.7.11
        next-hop-interface: enp1s0
        table-id: 254
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: ocp4-ipi-osp-worker-01
spec:
  online: false
  bootMode: legacy 
  # externallyProvisioned: true
  # hardwareProfile: unknown
  bootMACAddress: $(cat /data/install/mac.list.* | grep worker-01 | awk '{print $2}')
  bmc:
    address: redfish-virtualmedia://192.168.7.103:8000/redfish/v1/Systems/$(cat /data/install/vm.list.* | grep worker-01 | awk '{print $1}')
    credentialsName: worker-1-bmc-secret
    disableCertificateVerification: true
  rootDeviceHints:
    deviceName: /dev/vda
  preprovisioningNetworkDataName: ocp4-ipi-osp-worker-01-network-config-secret
EOF
oc -n openshift-machine-api create -f ${BASE_DIR}/data/install/bmh-01.yaml

# oc delete -f ${BASE_DIR}/data/install/bmh.yaml -n openshift-machine-api 

# DO NOT USE, restore, delete the vm
# oc -n openshift-machine-api delete -f ${BASE_DIR}/data/install/bmh.yaml


# oc delete -f ${BASE_DIR}/data/install/bmh-03.yaml -n openshift-machine-api 

oc get bmh -n openshift-machine-api 
# NAME                     STATE                    CONSUMER                      ONLINE   ERROR   AGE
# ocp4-ipi-osp-master-01   externally provisioned   acm-demo-one-42z8b-master-0   true             3h23m
# ocp4-ipi-osp-master-02   externally provisioned   acm-demo-one-42z8b-master-1   true             3h23m
# ocp4-ipi-osp-master-03   externally provisioned   acm-demo-one-42z8b-master-2   true             3h23m
# ocp4-ipi-osp-worker-01   externally provisioned                                 true             54s

oc get machinesets -n openshift-machine-api
# NAME                          DESIRED   CURRENT   READY   AVAILABLE   AGE
# acm-demo-one-42z8b-worker-0   0         0                             3h25m

# oc get machinesets -n openshift-machine-api -o json | jq -r .items[0].metadata.name

# # 扩容worker到3副本，会触发worker-2的部署
# oc scale --replicas=1 machineset $(oc get machinesets -n openshift-machine-api -o json | jq -r .items[0].metadata.name) -n openshift-machine-api

# oc scale --replicas=0 machineset $(oc get machinesets -n openshift-machine-api -o json | jq -r .items[0].metadata.name) -n openshift-machine-api

```

# install nfs

我们装的ocp集群，要想使用复杂的业务场景，肯定是需要存储的，我们是home lab，肯定想选取一个轻量化的存储方案，红帽自己的ODF对资源要求比较高，那么我们就选择k8s sig的NFS方案，装一个NFS服务，到集群里面。这个方案的特点，是把集群里面的一个节点，变成存储节点，用这个节点的一个目录作为数据存储空间，应用/pod可以在集群里面的各个节点运行。总的来说，虽然简单，但是性能受限。

如果有其他的需求，可以考虑cnv的host-path方案，或者openEBS方案。

## add local volumn

我们弄一个local volumn，给k8s sig的NFS方案作为后端存储。

```bash

# go to master-03, this is as storage node
# create the storage path
mkdir -p /var/wzh-local-pv/
chcon -Rt container_file_t /var/wzh-local-pv/

# on helper
cat << EOF > ${BASE_DIR}/data/install/local-pv.yaml
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-volume
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-local-pv
spec:
  capacity:
    storage: 450Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-volume
  local:
    path: /var/wzh-local-pv/
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - one-master-03.acm-demo-one.redhat.ren
EOF
oc create --save-config -f ${BASE_DIR}/data/install/local-pv.yaml

# oc delete -f ${BASE_DIR}/data/install/local-pv.yaml

```

## install nfs based on local pv

接下来，我们直接用yaml的方式，部署这个NFS服务，我们已经定制好了k8s sig的NFS服务启动yaml，我们下载，然后修改一下参数就可以直接启动了。他会创建对应的role, deployment等参数信息。

```bash

oc create ns nfs-system

# oc project nfs-system

cd ${BASE_DIR}/data/install

export http_proxy="http://127.0.0.1:18801"
export https_proxy=${http_proxy}

wget -O nfs.all.yaml https://raw.githubusercontent.com/wangzheng422/nfs-ganesha-server-and-external-provisioner/wzh/deploy/openshift/nfs.all.local.pv.yaml

unset http_proxy
unset https_proxy

/bin/cp -f nfs.all.yaml nfs.all.yaml.run

# sed -i 's/storageClassName: odf-lvm-vg1/storageClassName: local-volume/' nfs.all.yaml.run
sed -i 's/one-master-03.acm-demo-one.redhat.ren/one-master-03.acm-demo-one.redhat.ren/' nfs.all.yaml.run
sed -i 's/storage: 5Gi/storage: 450Gi/' nfs.all.yaml.run

oc create --save-config -n nfs-system -f nfs.all.yaml.run

# oc delete -n nfs-system -f nfs.all.yaml.run

```

# install cnv, nmstat, sriov operator

我们按照openstack operator的官方文档，安装几个依赖的operator，他们是
1. cnv， 这个是在openshift集群里面启动kvm虚拟机的，openstack的overcloud controller是用cnv启动的kvm来承载运行。
2. nmstat， 这个是配置openshift节点网卡参数的插件，openstack会定义很复杂的网络参数，从上面的架构图就能看出来。注意，nmstat只能修改ocp集群管理的节点网卡参数，对于已经更改了基础镜像，变成osp纳管的节点，这个插件是管不到的。
3. sriov,  这个是配置网卡直通的，作者还不太确定他用在什么地方，目前看，好像是cnv启动kvm的时候，会通过sriov把一堆网卡直通到kvm里面，干的是这个事情。

我们安装的时候，先用automatic approve的方式来安装，这样就能省去点击确认授权的步骤，装完了以后，我们在修改成manual approve的方式，防止集群自动升级operator。自动升级这个功能是很好，但是对于已经装好的集群，如果自动升级了，你还不知道，很可能就升级失败，导致你的集群功能受到影响。

## install cnv

我们先装CNV，这个是在ocp里面启动虚拟机用的。CNV会带起来一大堆的pod，所以安装的时间有点长。

```bash

# install cnv
cat << EOF > ${BASE_DIR}/data/install/cnv.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: openshift-cnv
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: kubevirt-hyperconverged-group
  namespace: openshift-cnv
spec:
  targetNamespaces:
    - openshift-cnv
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: hco-operatorhub
  namespace: openshift-cnv
spec:
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  name: kubevirt-hyperconverged
  startingCSV: kubevirt-hyperconverged-operator.v4.11.0
  channel: "stable" 
EOF
oc create --save-config -f ${BASE_DIR}/data/install/cnv.yaml

# oc delete -f ${BASE_DIR}/data/install/cnv.yaml

oc get csv -n openshift-cnv
# NAME                                       DISPLAY                    VERSION   REPLACES                                   PHASE
# kubevirt-hyperconverged-operator.v4.11.0   OpenShift Virtualization   4.11.0    kubevirt-hyperconverged-operator.v4.10.5   Succeeded

cat << EOF > ${BASE_DIR}/data/install/patch.yaml
spec:
  installPlanApproval: Manual
EOF
oc patch -n openshift-cnv subscription/hco-operatorhub --type merge --patch-file=${BASE_DIR}/data/install/patch.yaml


cat << EOF > ${BASE_DIR}/data/install/cnv-hc.yaml
apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
EOF
oc create --save-config -f ${BASE_DIR}/data/install/cnv-hc.yaml

# cat << EOF > ${BASE_DIR}/data/install/hostpath.yaml
# apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
# kind: HostPathProvisioner
# metadata:
#   name: hostpath-provisioner
# spec:
#   imagePullPolicy: IfNotPresent
#   storagePools: 
#   - name: wzh-cnv-hostpath-storage-pool
#     path: "/var/wzh-cnv-hostpath" 
# workload:
#   nodeSelector:
#     kubernetes.io/os: linux
# EOF

# oc create --save-config -f ${BASE_DIR}/data/install/hostpath.yaml

# cat << EOF > ${BASE_DIR}/data/install/sc.yaml
# apiVersion: storage.k8s.io/v1
# kind: StorageClass
# metadata:
#   name: hostpath-csi 
# provisioner: kubevirt.io.hostpath-provisioner
# reclaimPolicy: Delete 
# # volumeBindingMode: WaitForFirstConsumer 
# volumeBindingMode: Immediate
# parameters:
#   storagePool: wzh-cnv-hostpath-storage-pool
# EOF

# oc create --save-config -f ${BASE_DIR}/data/install/sc.yaml

# oc delete -f ${BASE_DIR}/data/install/sc.yaml
```

## install nmstat

接下来装nmstat，这个是设置网卡用的，记得不要设置ocp集群使用的通讯网口。

```bash

cat << EOF > ${BASE_DIR}/data/install/nmstat.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: openshift-nmstate
    name: openshift-nmstate
  name: openshift-nmstate
spec:
  finalizers:
  - kubernetes
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  annotations:
    olm.providedAPIs: NMState.v1.nmstate.io
  generateName: openshift-nmstate-
  name: openshift-nmstate-wzh
  namespace: openshift-nmstate
spec:
  targetNamespaces:
  - openshift-nmstate
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  labels:
    operators.coreos.com/kubernetes-nmstate-operator.openshift-nmstate: ""
  name: kubernetes-nmstate-operator
  namespace: openshift-nmstate
spec:
  channel: "4.11"
  name: kubernetes-nmstate-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF
oc create --save-config -f ${BASE_DIR}/data/install/nmstat.yaml

oc get csv -n openshift-nmstate
# NAME                                              DISPLAY                       VERSION               REPLACES   PHASE
# kubernetes-nmstate-operator.4.11.0-202210250857   Kubernetes NMState Operator   4.11.0-202210250857              Succeeded

cat << EOF > ${BASE_DIR}/data/install/patch.yaml
spec:
  installPlanApproval: Manual
EOF
oc patch -n openshift-nmstate subscription/kubernetes-nmstate-operator --type merge --patch-file=${BASE_DIR}/data/install/patch.yaml


cat << EOF > ${BASE_DIR}/data/install/nmstat-stat.yaml
---
apiVersion: nmstate.io/v1
kind: NMState
metadata:
  name: nmstate
EOF
oc create --save-config -f ${BASE_DIR}/data/install/nmstat-stat.yaml

```

## install sriov

最后，装sriov，这个是给cnv启动的kvm配置网口直通的。

```bash
# oc annotate ns/openshift-sriov-network-operator workload.openshift.io/allowed=management

cat << EOF > ${BASE_DIR}/data/install/sriov.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: openshift-sriov-network-operator
  annotations:
    workload.openshift.io/allowed: management
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: sriov-network-operators
  namespace: openshift-sriov-network-operator
spec:
  targetNamespaces:
  - openshift-sriov-network-operator
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sriov-network-operator-subscription
  namespace: openshift-sriov-network-operator
spec:
  channel: "4.11"
  name: sriov-network-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF
oc create --save-config -f ${BASE_DIR}/data/install/sriov.yaml

oc get csv -n openshift-sriov-network-operator
# NAME                                         DISPLAY                   VERSION               REPLACES   PHASE
# sriov-network-operator.4.11.0-202210250857   SR-IOV Network Operator   4.11.0-202210250857              Succeeded

oc get subscription -n openshift-sriov-network-operator
# NAME                                  PACKAGE                  SOURCE             CHANNEL
# sriov-network-operator-subscription   sriov-network-operator   redhat-operators   4.11

oc get subscription/sriov-network-operator-subscription -n openshift-sriov-network-operator -o json | jq .spec
# {
#   "channel": "4.11",
#   "name": "sriov-network-operator",
#   "source": "redhat-operators",
#   "sourceNamespace": "openshift-marketplace"
# }

cat << EOF > ${BASE_DIR}/data/install/patch.yaml
spec:
  installPlanApproval: Manual
EOF
oc patch -n openshift-sriov-network-operator subscription/sriov-network-operator-subscription --type merge --patch-file=${BASE_DIR}/data/install/patch.yaml

```

# install osp operator

我们马上就要开始安装openstack组件啦。我们参考的文档是[官方文档](https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html-single/rhosp_director_operator_for_openshift_container_platform/index#proc_creating-a-data-volume-for-the-base-operating-system_assembly_director-operator-deployment-scenario-overcloud-with-hyper-converged-storage-hci)，官方文档，写的已经很用心，很好了，但是还是免不了有错误。我们会在接下来的步骤中，修正文档里面的错误。

## build operator images

安装的第一步，居然是自己编译osp operator的镜像？算了，毕竟是TP版本的软件，有一些不完善，也能理解。根据文档，我们需要找最新的版本，自己打包operator catalog 镜像，这个镜像是一个operator hub的catalog，可以简单理解为，我们在ocp的应用商店里面，开了一个新的门面，叫openstack，里面就有一样商品，叫openstack。

```bash

# https://github.com/openstack-k8s-operators/osp-director-operator
# [osp-director-operator](https://catalog.redhat.com/software/containers/rhosp-rhel8-tech-preview/osp-director-operator/607dd3bf87c834779d77611b) 
# [osp-director-operator-bundle](https://catalog.redhat.com/software/containers/rhosp-rhel8-tech-preview/osp-director-operator-bundle/607dd43903f4b3563ab483b3)

#########################
# on helper
# run as root
cd /data/ocp4/4.11.6/
tar zvxf opm-linux-4.11.6.tar.gz
install opm /usr/local/bin/

/bin/cp -f /etc/containers/policy.json /etc/containers/policy.json.bak
cat << EOF > /etc/containers/policy.json
{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ],
    "transports":
        {
            "docker-daemon":
                {
                    "": [{"type":"insecureAcceptAnything"}]
                }
        }
}
EOF

# end run as root
#########################

# registry.redhat.io/rhosp-rhel8-tech-preview/osp-director-operator-bundle:1.2.3-12
BUNDLE_IMG="registry.redhat.io/rhosp-rhel8-tech-preview/osp-director-operator-bundle:1.2.3-12"
INDEX_IMG="quay.io/wangzheng422/osp-director-operator-index:1.2.3-12"
opm index add --bundles ${BUNDLE_IMG} --tag ${INDEX_IMG} -u podman --pull-tool podman
podman push ${INDEX_IMG}

```

## install openstack director operator

编译好了openstack的catalog镜像，我们就用这个镜像，部署一个catalog，并且安装openstack director operator.

```bash
oc new-project openstack

cat << EOF > ${BASE_DIR}/data/install/osp-director-operator.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: osp-director-operator-index
  namespace: openstack
spec:
  sourceType: grpc
  # image: quay.io/openstack-k8s-operators/osp-director-operator-index:1.0.0-1
  # image: quay.io/openstack-k8s-operators/osp-director-operator-index:1.2.3
  image: quay.io/wangzheng422/osp-director-operator-index@sha256:ac810497a3b29662573e0843715285a1ad69e3fe7a8c7b6e5fe43d2f6d5bda8d
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: "osp-director-operator-group"
  namespace: openstack
spec:
  targetNamespaces:
  - openstack
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: osp-director-operator-subscription
  namespace: openstack
spec:
  config:
    env:
    - name: WATCH_NAMESPACE
      value: openstack,openshift-machine-api,openshift-sriov-network-operator
  source: osp-director-operator-index
  sourceNamespace: openstack
  name: osp-director-operator
EOF

oc create --save-config -f ${BASE_DIR}/data/install/osp-director-operator.yaml

# oc delete -f ${BASE_DIR}/data/install/osp-director-operator.yaml

oc get operators
# NAME                                                      AGE
# kubernetes-nmstate-operator.openshift-nmstate             21h
# kubevirt-hyperconverged.openshift-cnv                     22h
# osp-director-operator.openstack                           17m
# sriov-network-operator.openshift-sriov-network-operator   21h

oc get csv -n openstack
# NAME                           DISPLAY                 VERSION   REPLACES   PHASE
# osp-director-operator.v1.2.3   OSP Director Operator   1.2.3                Succeeded

```

# try to deploy osp

有了openstack director operator，我们就要真正的开始一步一步的安装一个openstack overcloud啦。

我们参考的文档在这里: [Chapter 7. Director operator deployment scenario: Overcloud with Hyper-Converged Infrastructure (HCI)](https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html-single/rhosp_director_operator_for_openshift_container_platform/index#assembly_director-operator-deployment-scenario-overcloud-with-hyper-converged-infrastructure-hci_rhosp-director-operator)

## upload rhel image

openstack是虚机平台，我们需要准备操作系统镜像，我们就下载官网的rhel8.6镜像，并且按照文档的要求，进行小小的定制化。

然后用cnv的命令行virtctl，去把这个镜像上传。

```bash

# download rhel-8.6-x86_64-kvm.qcow2 from redhat website
ls -l /data/down | grep rhel
# -rw-r--r--. 1 root root 8770508800 Apr 27  2022 rhel-8.6-aarch64-dvd.iso
# -rw-r--r--. 1 root root  832438272 May 10 13:23 rhel-8.6-x86_64-kvm.qcow2

export PROXY="http://127.0.0.1:18801"

subscription-manager repos --proxy=$PROXY --enable=cnv-4.11-for-rhel-8-x86_64-rpms

dnf install -y kubevirt-virtctl libguestfs-tools-c

/bin/cp -f /data/down/rhel-8.6-x86_64-kvm.qcow2 /data/down/rhel-8.6-x86_64-kvm-wzh.qcow2

virt-customize -a /data/down/rhel-8.6-x86_64-kvm-wzh.qcow2 --run-command 'sed -i -e "s/^\(kernelopts=.*\)net.ifnames=0 \(.*\)/\1\2/" /boot/grub2/grubenv'
virt-customize -a /data/down/rhel-8.6-x86_64-kvm-wzh.qcow2 --run-command 'sed -i -e "s/^\(GRUB_CMDLINE_LINUX=.*\)net.ifnames=0 \(.*\)/\1\2/" /etc/default/grub'

virtctl version
# Client Version: version.Info{GitVersion:"v0.36.5-2-gdd266dff9", GitCommit:"dd266dff95f7de9f79e3e0e5d4867c5ba9d50c9d", GitTreeState:"clean", BuildDate:"2022-04-01T22:51:18Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}
# dial tcp [::1]:8080: connect: connection refused

# copy qcow2 to helper
scp /data/down/rhel-8.6-x86_64-kvm-wzh.qcow2    root@192.168.7.11:/data/swap/

# on helper download virtctl 
export http_proxy="http://127.0.0.1:18801"
export https_proxy=${http_proxy}

export VERSION=v0.53.2
wget https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/virtctl-${VERSION}-linux-amd64

install -m 755 virtctl-${VERSION}-linux-amd64 /usr/local/bin/virtctl

unset http_proxy
unset https_proxy

su - 3nodeipi

oc get storageclass
# NAME             PROVISIONER                        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
# hostpath-csi     kubevirt.io.hostpath-provisioner   Delete          Immediate           false                  8m36s
# redhat-ren-nfs   redhat.ren/nfs                     Delete          Immediate           false                  3m27s

virtctl image-upload dv openstack-base-img -n openstack --size=50Gi --image-path=/data/swap/rhel-8.6-x86_64-kvm-wzh.qcow2  --storage-class redhat-ren-nfs --access-mode ReadWriteOnce --insecure
# PVC openstack/openstack-base-img not found
# DataVolume openstack/openstack-base-img created
# Waiting for PVC openstack-base-img upload pod to be ready...
# Pod now ready
# Uploading data to https://cdi-uploadproxy-openshift-cnv.apps.acm-demo-one.redhat.ren

#  797.50 MiB / 797.50 MiB [===============================================================================================================================================================] 100.00% 13s

# Uploading data completed successfully, waiting for processing to complete, you can hit ctrl-c without interrupting the progress
# Processing completed successfully
# Uploading rhel-8.6-x86_64-kvm-wzh.qcow2 completed successfully

# virtctl image-upload dv openstack-base-img -n openstack --no-create --size=50Gi --image-path=/data/swap/rhel-8.6-x86_64-kvm-wzh.qcow2  --storage-class redhat-ren-nfs --access-mode ReadWriteOnce --insecure

oc get datavolume
# NAME                 PHASE         PROGRESS   RESTARTS   AGE
# openstack-base-img   UploadReady   N/A        1          113s

# in some case, import fail, just delete the data volume to restart
# oc delete datavolume/openstack-base-img

# ensure there is only one pvc
oc get pv
# NAME               CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
# example-local-pv   450Gi      RWO            Retain           Bound    nfs-system/lvm-file-pvc   local-volume            42m

# in some case, import will never success, 
# it is because cdi is kill by OOM, the reason is un-knonw.
# just reboot master-03 to fix that.

```

## config key for git service, define default password

接下来，我们导入git服务的密钥，后面openstack会把安装脚本上传。

然后我们还要设置主机默认的用户名和口令。

```bash
oc create secret generic git-secret -n openstack --from-file=git_ssh_identity=${BASE_DIR}/.ssh/id_rsa --from-literal=git_url=ssh://git@quaylab.infra.redhat.ren:10022/root/openstack.git

# Setting the root password for nodes
echo -n "redhat" | base64
# cmVkaGF0

cat << EOF > ${BASE_DIR}/data/install/openstack-userpassword.yaml
apiVersion: v1
kind: Secret
metadata:
  name: userpassword
  namespace: openstack
data:
  NodeRootPassword: "`echo -n "redhat" | base64`"
EOF

oc create --save-config -f ${BASE_DIR}/data/install/openstack-userpassword.yaml -n openstack
```

## define network parameter

我们定义openstack用到的网络参数。这里面很绕，因为这个定义里面，IP地址的配置，是openstack的controller, computer节点都使用的。但是bridge, bridge对应的网卡，network附着的bridge这些配置，只是对openshift节点有效。

总的来说，这个网络参数配置，是针对openshift节点的，虽然他的名字是OpenStackNetConfig

这一步操作，对应到架构图，是这部分：

![](dia/ocp.osp.s85.drawio.svg)

```bash
# network link name no longer than 15
# https://access.redhat.com/solutions/2425471
# https://github.com/openstack-k8s-operators/osp-director-dev-tools/blob/osp16_tech_preview/ansible/templates/osp/tripleo_heat_envs/vlan/network-environment.yaml.j2
# https://github.com/openstack-k8s-operators/osp-director-dev-tools/blob/master/ansible/templates/osp/netconfig/osnetconfig.yaml.j2

cat << EOF > ${BASE_DIR}/data/install/openstacknetconfig.yaml
apiVersion: osp-director.openstack.org/v1beta1
kind: OpenStackNetConfig
metadata:
  name: openstacknetconfig
spec:
  attachConfigurations:
    br-osp:
      nodeNetworkConfigurationPolicy:
        nodeSelector:
          node-role.kubernetes.io/master: ""
        desiredState:
          interfaces:
          - bridge:
              options:
                stp:
                  enabled: false
              port:
              - name: enp4s0
            description: Linux bridge with enp4s0 as a port
            name: br-osp
            state: up
            type: linux-bridge
            mtu: 1500
    br-osp-ex:
      nodeNetworkConfigurationPolicy:
        nodeSelector:
          node-role.kubernetes.io/master: ""
        desiredState:
          interfaces:
          - bridge:
              options:
                stp:
                  enabled: false
              port:
              - name: enp3s0
            description: Linux bridge with enp3s0 as a port
            name: br-osp-ex
            state: up
            type: linux-bridge
            mtu: 1500

  # optional DnsServers list
  dnsServers:
  - 192.168.7.11
  # optional DnsSearchDomains list
  dnsSearchDomains:
  - osp-demo.redhat.ren
  - some.other.domain
  # DomainName of the OSP environment
  domainName: osp-demo.redhat.ren
  networks:
  - name: Control
    nameLower: ctlplane
    subnets:
    - name: ctlplane
      ipv4:
        allocationEnd: 192.168.7.60
        allocationStart: 192.168.7.40
        cidr: 192.168.7.0/24
        gateway: 192.168.7.11
      attachConfiguration: br-osp
  - name: InternalApi
    nameLower: internal_api
    mtu: 1350
    subnets:
    - name: internal_api
      attachConfiguration: br-osp
      vlan: 20
      ipv4:
        allocationEnd: 172.17.0.250
        allocationStart: 172.17.0.10
        cidr: 172.17.0.0/24
  - name: External
    nameLower: external
    subnets:
    - name: external
      ipv4:
        allocationEnd: 172.21.6.60
        allocationStart: 172.21.6.40
        cidr: 172.21.6.0/24
        gateway: 172.21.6.254
      attachConfiguration: br-osp-ex
  - name: Storage
    nameLower: storage
    mtu: 1500
    subnets:
    - name: storage
      ipv4:
        allocationEnd: 172.18.0.250
        allocationStart: 172.18.0.10
        cidr: 172.18.0.0/24
      vlan: 30
      attachConfiguration: br-osp
  - name: StorageMgmt
    nameLower: storage_mgmt
    mtu: 1500
    subnets:
    - name: storage_mgmt
      ipv4:
        allocationEnd: 172.19.0.250
        allocationStart: 172.19.0.10
        cidr: 172.19.0.0/24
      vlan: 40
      attachConfiguration: br-osp
  - name: Tenant
    nameLower: tenant
    vip: False
    mtu: 1500
    subnets:
    - name: tenant
      ipv4:
        allocationEnd: 172.20.0.250
        allocationStart: 172.20.0.10
        cidr: 172.20.0.0/24
      vlan: 50
      attachConfiguration: br-osp
EOF

oc create --save-config -f ${BASE_DIR}/data/install/openstacknetconfig.yaml -n openstack

# oc delete -f ${BASE_DIR}/data/install/openstacknetconfig.yaml -n openstack
# oc apply -f ${BASE_DIR}/data/install/openstacknetconfig.yaml -n openstack

oc get openstacknetconfig/openstacknetconfig -n openstack
# NAME                 ATTACHCONFIG DESIRED   ATTACHCONFIG READY   NETWORKS DESIRED   NETWORKS READY   PHYSNETWORKS DESIRED   PHYSNETWORKS READY   STATUS       REASON
# openstacknetconfig   2                      2                    6                  6                1                      1                    Configured   OpenStackNetConfig openstacknetconfig all resources configured

# oc get openstacknetattach -n openstack

oc get openstacknet -n openstack
# NAME          CIDR             DOMAIN                            MTU    VLAN   VIP     GATEWAY        ROUTES   RESERVED IPS   STATUS
# ctlplane      192.168.7.0/24   ctlplane.osp-demo.redhat.ren      1500   0      true    192.168.7.11   []       0              Configured
# external      172.21.6.0/24    external.osp-demo.redhat.ren      1500   0      true    172.21.6.254   []       0              Configured
# internalapi   172.17.0.0/24    internalapi.osp-demo.redhat.ren   1350   20     true                   []       0              Configured
# storage       172.18.0.0/24    storage.osp-demo.redhat.ren       1500   30     true                   []       0              Configured
# storagemgmt   172.19.0.0/24    storagemgmt.osp-demo.redhat.ren   1500   40     true                   []       0              Configured
# tenant        172.20.0.0/24    tenant.osp-demo.redhat.ren        1500   50     false                  []       0              Configured

oc get network-attachment-definitions -n openstack
# NAME                 AGE
# ctlplane             2m12s
# ctlplane-static      2m11s
# external             2m11s
# external-static      2m11s
# internalapi          2m11s
# internalapi-static   2m11s
# storage              2m11s
# storage-static       2m11s
# storagemgmt          2m10s
# storagemgmt-static   2m10s
# tenant               2m10s
# tenant-static        2m10s

oc get nncp
# NAME        STATUS      REASON
# br-osp      Available   SuccessfullyConfigured
# br-osp-ex   Available   SuccessfullyConfigured

```

## deploy controller

我们定义一个单节点的controller，这个定义保存后，openshift会通过cnv启动一个kvm，这个kvm会使用我们之前上传的rhel镜像作为操作系统，启动完成以后，就以一个空的操作系统，静静的运行在那里。

同时，他会运行一个openstack client的pod，我们后面日常对openstack的操作，就都会在这个openstack client里面。

注意，这里面文档有bug。文档里面的版本是v1beta2，而我们的镜像里面只有v1beta1，所以我们需要对配置做一些小的调整。

这一步操作，对应到架构图，是这部分：

![](dia/ocp.osp.s90.drawio.svg)

```bash
# here version mismatch with official document.
# we use old official document, which can't be found. :(
cat << EOF > ${BASE_DIR}/data/install/openstack-controller.yaml
apiVersion: osp-director.openstack.org/v1beta1
kind: OpenStackControlPlane
metadata:
  name: overcloud
  namespace: openstack
spec:
  # openStackClientImageURL: registry.redhat.io/rhosp-beta/openstack-tripleoclient:16.2
  openStackClientNetworks:
        - ctlplane
        - external
        - internal_api
  openStackClientStorageClass: redhat-ren-nfs
  passwordSecret: userpassword
  gitSecret: git-secret
  virtualMachineRoles:
    controller:
      roleName: Controller
      roleCount: 1
      networks:
        - ctlplane
        - internal_api
        - external
        - tenant
        - storage
        - storage_mgmt
      cores: 6
      memory: 12
      diskSize: 60
      baseImageVolumeName: openstack-base-img
      storageClass: redhat-ren-nfs
EOF
oc create --save-config -f ${BASE_DIR}/data/install/openstack-controller.yaml -n openstack

# oc delete -f ${BASE_DIR}/data/install/openstack-controller.yaml -n openstack
# oc apply -f ${BASE_DIR}/data/install/openstack-controller.yaml -n openstack

# here, it will take a long time, because it will clone pvc to 3 pvc
# half to 1 hour, based on your disk performance.

oc get openstackcontrolplane/overcloud -n openstack
# NAME        VMSETS DESIRED   VMSETS READY   CLIENT READY   STATUS        REASON
# overcloud   1                1              true           Provisioned   All requested OSVMSets have been provisioned

oc get openstackcontrolplane -n openstack
# NAME        VMSETS DESIRED   VMSETS READY   CLIENT READY   STATUS        REASON
# overcloud   1                1              true           Provisioned   All requested OSVMSets have been provisioned

oc get openstackvmsets -n openstack
# NAME         DESIRED   READY   STATUS        REASON
# controller   3         3       Provisioned   All requested VirtualMachines have been provisioned

oc get virtualmachines -n openstack
# NAME           AGE    STATUS    READY
# controller-0   107m   Running   True
# controller-1   107m   Running   True
# controller-2   107m   Running   True

```

![](imgs/2022-11-21-22-10-46.png)

## define openstack install script

接着，我们按照官方文档，定义openstack install script，这个安装脚本，是配置computer节点网络的。

官方文档里面有个bug，就是没定义StorageMgmt网络，我们补充进去就好了。

安装脚本的定义分好几个步骤，有几个步骤，是把官方文档copy past进去，还有步骤，是在openstack client pod里面，创建模板，然后导出，总之，按照步骤做就好，并不难。

```bash

# on helper
mkdir -p ${BASE_DIR}/data/custom_templates
mkdir -p ${BASE_DIR}/data/custom_environment_files

/bin/rm -rf ${BASE_DIR}/data/custom_templates/*
/bin/rm -rf ${BASE_DIR}/data/custom_environment_files/*

cat << 'EOF' > ${BASE_DIR}/data/custom_templates/net-config-two-nic-vlan-computehci.yaml
heat_template_version: rocky
description: >
  Software Config to drive os-net-config to configure VLANs for the Compute role.
parameters:
  ControlPlaneIp:
    default: ''
    description: IP address/subnet on the ctlplane network
    type: string
  ControlPlaneSubnetCidr:
    default: ''
    description: >
      The subnet CIDR of the control plane network. (The parameter is
      automatically resolved from the ctlplane subnet's cidr attribute.)
    type: string
  ControlPlaneDefaultRoute:
    default: ''
    description: The default route of the control plane network. (The parameter
      is automatically resolved from the ctlplane subnet's gateway_ip attribute.)
    type: string
  ControlPlaneStaticRoutes:
    default: []
    description: >
      Routes for the ctlplane network traffic.
      JSON route e.g. [{'destination':'10.0.0.0/16', 'nexthop':'10.0.0.1'}]
      Unless the default is changed, the parameter is automatically resolved
      from the subnet host_routes attribute.
    type: json
  ControlPlaneMtu:
    default: 1500
    description: The maximum transmission unit (MTU) size(in bytes) that is
      guaranteed to pass through the data path of the segments in the network.
      (The parameter is automatically resolved from the ctlplane network's mtu attribute.)
    type: number
  StorageIpSubnet:
    default: ''
    description: IP address/subnet on the storage network
    type: string
  StorageNetworkVlanID:
    default: 30
    description: Vlan ID for the storage network traffic.
    type: number
  StorageMtu:
    default: 1500
    description: The maximum transmission unit (MTU) size(in bytes) that is
      guaranteed to pass through the data path of the segments in the
      Storage network.
    type: number
  StorageInterfaceRoutes:
    default: []
    description: >
      Routes for the storage network traffic.
      JSON route e.g. [{'destination':'10.0.0.0/16', 'nexthop':'10.0.0.1'}]
      Unless the default is changed, the parameter is automatically resolved
      from the subnet host_routes attribute.
    type: json
  StorageMgmtIpSubnet:
    default: ''
    description: IP address/subnet on the storage_mgmt network
    type: string
  StorageMgmtNetworkVlanID:
    default: 40
    description: Vlan ID for the storage_mgmt network traffic.
    type: number
  StorageMgmtMtu:
    default: 1500
    description: The maximum transmission unit (MTU) size(in bytes) that is
      guaranteed to pass through the data path of the segments in the
      StorageMgmt network.
    type: number
  StorageMgmtInterfaceRoutes:
    default: []
    description: >
      Routes for the storage_mgmt network traffic.
      JSON route e.g. [{'destination':'10.0.0.0/16', 'nexthop':'10.0.0.1'}]
      Unless the default is changed, the parameter is automatically resolved
      from the subnet host_routes attribute.
    type: json
  InternalApiIpSubnet:
    default: ''
    description: IP address/subnet on the internal_api network
    type: string
  InternalApiNetworkVlanID:
    default: 20
    description: Vlan ID for the internal_api network traffic.
    type: number
  InternalApiMtu:
    default: 1500
    description: The maximum transmission unit (MTU) size(in bytes) that is
      guaranteed to pass through the data path of the segments in the
      InternalApi network.
    type: number
  InternalApiInterfaceRoutes:
    default: []
    description: >
      Routes for the internal_api network traffic.
      JSON route e.g. [{'destination':'10.0.0.0/16', 'nexthop':'10.0.0.1'}]
      Unless the default is changed, the parameter is automatically resolved
      from the subnet host_routes attribute.
    type: json
  TenantIpSubnet:
    default: ''
    description: IP address/subnet on the tenant network
    type: string
  TenantNetworkVlanID:
    default: 50
    description: Vlan ID for the tenant network traffic.
    type: number
  TenantMtu:
    default: 1500
    description: The maximum transmission unit (MTU) size(in bytes) that is
      guaranteed to pass through the data path of the segments in the
      Tenant network.
    type: number
  TenantInterfaceRoutes:
    default: []
    description: >
      Routes for the tenant network traffic.
      JSON route e.g. [{'destination':'10.0.0.0/16', 'nexthop':'10.0.0.1'}]
      Unless the default is changed, the parameter is automatically resolved
      from the subnet host_routes attribute.
    type: json
  ExternalMtu:
    default: 1500
    description: The maximum transmission unit (MTU) size(in bytes) that is
      guaranteed to pass through the data path of the segments in the
      External network.
    type: number
  DnsServers: # Override this via parameter_defaults
    default: []
    description: >
      DNS servers to use for the Overcloud (2 max for some implementations).
      If not set the nameservers configured in the ctlplane subnet's
      dns_nameservers attribute will be used.
    type: comma_delimited_list
  DnsSearchDomains: # Override this via parameter_defaults
    default: []
    description: A list of DNS search domains to be added (in order) to resolv.conf.
    type: comma_delimited_list

resources:

  MinViableMtu:
    # This resource resolves the minimum viable MTU for interfaces, bonds and
    # bridges that carry multiple VLANs. Each VLAN may have different MTU. The
    # bridge, bond or interface must have an MTU to allow the VLAN with the
    # largest MTU.
    type: OS::Heat::Value
    properties:
      type: number
      value:
        yaql:
          expression: $.data.max()
          data:
            - {get_param: ControlPlaneMtu}
            - {get_param: StorageMtu}
            - {get_param: StorageMgmtMtu}
            - {get_param: InternalApiMtu}
            - {get_param: TenantMtu}
            - {get_param: ExternalMtu}

  OsNetConfigImpl:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template:
            get_file: /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh
          params:
            $network_config:
              network_config:
              - type: interface
                name: nic4
                mtu:
                  get_attr: [MinViableMtu, value]
                use_dhcp: false
                dns_servers:
                  get_param: DnsServers
                domain:
                  get_param: DnsSearchDomains
                addresses:
                - ip_netmask:
                    list_join:
                    - /
                    - - get_param: ControlPlaneIp
                      - get_param: ControlPlaneSubnetCidr
                routes:
                  list_concat_unique:
                    - get_param: ControlPlaneStaticRoutes
                    - - default: true
                        next_hop:
                          get_param: ControlPlaneDefaultRoute
              - type: vlan
                mtu:
                  get_param: StorageMtu
                device: nic4
                vlan_id:
                  get_param: StorageNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: StorageIpSubnet
                routes:
                  list_concat_unique:
                    - get_param: StorageInterfaceRoutes
              - type: vlan
                device: nic4
                mtu:
                  get_param: StorageMgmtMtu
                vlan_id:
                  get_param: StorageMgmtNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: StorageMgmtIpSubnet
                routes:
                  list_concat_unique:
                    - get_param: StorageMgmtInterfaceRoutes                    
              - type: vlan
                mtu:
                  get_param: InternalApiMtu
                device: nic4
                vlan_id:
                  get_param: InternalApiNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: InternalApiIpSubnet
                routes:
                  list_concat_unique:
                    - get_param: InternalApiInterfaceRoutes

              - type: ovs_bridge
                # This will default to br-ex, anything else   requires specific
                # bridge mapping entries for it to be used.
                name: bridge_name
                mtu:
                  get_param: ExternalMtu
                use_dhcp: false
                members:
                - type: interface
                  name: nic3
                  mtu:
                    get_param: ExternalMtu
                  use_dhcp: false
                  primary: true
                - type: vlan
                  mtu:
                    get_param: TenantMtu
                  vlan_id:
                    get_param: TenantNetworkVlanID
                  addresses:
                  - ip_netmask:
                      get_param: TenantIpSubnet
                  routes:
                    list_concat_unique:
                      - get_param: TenantInterfaceRoutes
outputs:
  OS::stack_id:
    description: The OsNetConfigImpl resource.
    value:
      get_resource: OsNetConfigImpl
EOF


oc rsh -n openstack openstackclient
# in the shell
unset OS_CLOUD
cd /home/cloud-admin/
openstack overcloud roles generate Controller ComputeHCI > roles_data.yaml
exit

oc cp openstack/openstackclient:home/cloud-admin/roles_data.yaml ${BASE_DIR}/data/custom_templates/roles_data.yaml

cd ${BASE_DIR}/data/custom_templates
tar -cvzf custom-config.tar.gz *.yaml
oc delete configmap tripleo-tarball-config -n openstack
oc create configmap tripleo-tarball-config --from-file=custom-config.tar.gz -n openstack

oc get configmap/tripleo-tarball-config -n openstack
# NAME                     DATA   AGE
# tripleo-tarball-config   1      7s

cat << EOF > ${BASE_DIR}/data/custom_environment_files/network-environment.yaml
resource_registry:
  OS::TripleO::ComputeHCI::Net::SoftwareConfig: net-config-two-nic-vlan-computehci.yaml

# parameter_defaults:
#   # self define
#   NeutronBridgeMappings: datacentre:br-osp-ex
EOF

cat << EOF > ${BASE_DIR}/data/custom_environment_files/compute-hci.yaml
resource_registry:
  OS::TripleO::Services::CephMgr: deployment/ceph-ansible/ceph-mgr.yaml
  OS::TripleO::Services::CephMon: deployment/ceph-ansible/ceph-mon.yaml
  OS::TripleO::Services::CephOSD: deployment/ceph-ansible/ceph-osd.yaml
  OS::TripleO::Services::CephClient: deployment/ceph-ansible/ceph-client.yaml

parameter_defaults:
  # needed for now because of the repo used to create tripleo-deploy image
  CephAnsibleRepo: "rhelosp-ceph-4-tools"
  CephAnsiblePlaybookVerbosity: 3
  CinderEnableIscsiBackend: false
  CinderEnableRbdBackend: true
  CinderBackupBackend: ceph
  CinderEnableNfsBackend: false
  NovaEnableRbdBackend: true
  GlanceBackend: rbd
  CinderRbdPoolName: "volumes"
  NovaRbdPoolName: "vms"
  GlanceRbdPoolName: "images"
  CephPoolDefaultPgNum: 32
  CephPoolDefaultSize: 2
  CephAnsibleDisksConfig:
    devices:
      - '/dev/vdb'
      - '/dev/vdc'
      - '/dev/vdd'
    osd_scenario: lvm
    osd_objectstore: bluestore
  CephAnsibleExtraConfig:
    is_hci: true
  CephConfigOverrides:
    rgw_swift_enforce_content_length: true
    rgw_swift_versioning_enabled: true
EOF

oc delete configmap -n openstack heat-env-config 
oc create configmap -n openstack heat-env-config --from-file=${BASE_DIR}/data/custom_environment_files/ --dry-run=client -o yaml | oc apply -f -

oc get configmap/heat-env-config -n openstack
# NAME              DATA   AGE
# heat-env-config   2      4s

```

## define computer node

接下来，我们定义computer node。在定义computer node之前，我们openshift集群是有一个worker节点的，这个worker节点是空的，啥也没有，我们通过定义OpenStackBaremetalSet，调用openshift的metal3相关的功能，用我们指定的镜像，把这个worker节点刷成一个rhel节点。

这一步操作，对应到架构图，是这部分：

![](dia/ocp.osp.s95.drawio.svg)

```bash

cat << EOF > ${BASE_DIR}/data/install/openstack-hcicompute.yaml
apiVersion: osp-director.openstack.org/v1beta1
kind: OpenStackBaremetalSet
metadata:
  name: computehci
  namespace: openstack
spec:
  count: 1
  baseImageUrl: http://192.168.7.11:8080/rhel-8.6-x86_64-kvm-wzh.qcow2
  deploymentSSHSecret: osp-controlplane-ssh-keys
  ctlplaneInterface: enp4s0
  networks:
    - ctlplane
    - internal_api
    - tenant
    - storage
    - storage_mgmt
  roleName: ComputeHCI
  passwordSecret: userpassword
EOF
oc create --save-config -f ${BASE_DIR}/data/install/openstack-hcicompute.yaml -n openstack

# oc delete -f ${BASE_DIR}/data/install/openstack-hcicompute.yaml -n openstack

# very tricky, after read source code, there is a buggy logic to check the online to false.
# cat << EOF > ${BASE_DIR}/data/install/patch.yaml
# spec:
#   online: fales
# EOF
# oc patch -n openshift-machine-api BareMetalHost/ocp4-ipi-osp-worker-01 --type merge --patch-file=${BASE_DIR}/data/install/patch.yaml

# ssh into the worker-1, and add public access ip address
# so it can download ironic agent podman image
# and the ironic agent will write base image to disk
# but first, it will boot using coreos live iso
# ssh -i id_rsa core@172.22.0.199
# sudo -i
# nmcli con add ifname enp1s0 con-name enp1s0 type ethernet ipv4.method manual ipv4.addresses 192.168.7.26/24 ipv4.dns 192.168.7.11
# nmcli con up enp1s0

# /bin/qemu-img convert -O host_device -t directsync -S 0 -W /tmp/compressed-rhel-8.6-x86_64-kvm-wzh.qcow2 /dev/vda
# sgdisk -e /dev/vda
# sgdisk -Z /dev/vda3

oc describe crd openstackbaremetalset

oc get openstackbaremetalset -n openstack
# NAME         DESIRED   READY   STATUS        REASON
# computehci   1         1       Provisioned   All requested BaremetalHosts have been provisioned

oc get openstackbaremetalset/computehci -n openstack
# NAME         DESIRED   READY   STATUS        REASON
# computehci   1         1       Provisioned   All requested BaremetalHosts have been provisioned

oc get baremetalhosts -n openshift-machine-api
# NAME                     STATE                    CONSUMER                      ONLINE   ERROR   AGE
# ocp4-ipi-osp-master-01   externally provisioned   acm-demo-one-8zwdl-master-0   true             126m
# ocp4-ipi-osp-master-02   externally provisioned   acm-demo-one-8zwdl-master-1   true             126m
# ocp4-ipi-osp-master-03   externally provisioned   acm-demo-one-8zwdl-master-2   true             126m
# ocp4-ipi-osp-worker-01   provisioned              computehci                    true             54m

```

![](imgs/2022-11-21-22-11-37.png)

## patch for openstack nodes

在继续操作之前，我们需要给已有的openstack 节点，包括controller, computer，打一些配置上去，因为我们是离线环境，主要就是把repo yum源，还有container registry源，都指向内网的环境。

```bash
###########################
# add repo for osp nodes
oc rsh -n openstack openstackclient

cd /home/cloud-admin

VAR_URL=http://192.168.7.11:5000/osp.repo

# ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory ./rhsm.yaml

ansible -i /home/cloud-admin/ctlplane-ansible-inventory overcloud -a "sudo dnf config-manager --add-repo $VAR_URL"

ansible -i /home/cloud-admin/ctlplane-ansible-inventory overcloud -a "sudo mkdir -p /etc/cni/net.d"

scp root@192.168.7.11:/data/ocp4/image.registries.conf ./
sed -i 's/nexus.infra.redhat.ren/192.168.7.11/g' image.registries.conf 

ansible -i /home/cloud-admin/ctlplane-ansible-inventory overcloud -a "sudo mkdir -p /etc/containers/registries.conf.d/"
ansible -i /home/cloud-admin/ctlplane-ansible-inventory overcloud -b -m copy -a "src=./image.registries.conf dest=/etc/containers/registries.conf.d/image.registries.conf"

cat << EOF > ./policy.json
{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ],
    "transports":
        {
            "docker-daemon":
                {
                    "": [{"type":"insecureAcceptAnything"}]
                }
        }
}
EOF
ansible -i /home/cloud-admin/ctlplane-ansible-inventory overcloud -b -m copy -a "src=./policy.json dest=/etc/containers/policy.json"

```

## generate ansible playbooks

我们终于到了最后的步骤了，我们将要创建安装用的ansible playbook。

在这里面，有一个osp operator的小bug，因为我们的git ssh端口不是22端口，所以我们的git uri形式不是 git@server:root/openstack.git , 而是 ssh://git@quaylab.infra.redhat.ren:10022/root/openstack.git ，这就造成的脚本解析错误，暂时没有好办法解决，只能到pod里面去，把ssh config文件patch一下。

```bash
cat << EOF > ${BASE_DIR}/data/install/openstack-config-generator.yaml
apiVersion: osp-director.openstack.org/v1beta1
kind: OpenStackConfigGenerator
metadata:
  name: default
  namespace: openstack
spec:
  enableFencing: false
  gitSecret: git-secret
  # imageURL: registry.redhat.io/rhosp-rhel8/openstack-tripleoclient:16.2
  heatEnvConfigMap: heat-env-config
  # List of heat environment files to include from tripleo-heat-templates/environments
  # heatEnvs:
  # - ssl/tls-endpoints-public-dns.yaml
  # - ssl/enable-tls.yaml
  tarballConfigMap: tripleo-tarball-config
  # interactive: true
EOF
oc create --save-config -f ${BASE_DIR}/data/install/openstack-config-generator.yaml -n openstack

# oc delete -f ${BASE_DIR}/data/install/openstack-config-generator.yaml -n openstack

oc get openstackconfiggenerator/default -n openstack
# NAME      STATUS
# default   Initializing

# fix for ssh connect bugs
# if the git host on ssh other than port 22
# the osp script will buggy
watch oc get pod -l job-name=generate-config-default

oc rsh $(oc get pod -o name -l job-name=generate-config-default)
# ls -la /home/cloud-admin/
cat /home/cloud-admin/.ssh/config

cat << EOF >> /home/cloud-admin/.ssh/config

Host quaylab.infra.redhat.ren
    User root
    IdentityFile /home/cloud-admin/.ssh/git_id_rsa
    StrictHostKeyChecking no

EOF
# git clone ssh://git@quaylab.infra.redhat.ren:10022/root/openstack.git

oc get openstackconfiggenerator/default -n openstack
# NAME      STATUS
# default   Generating

```

## run ansible playbooks

我们将要在openstack client pod里面，运行ansible playbook脚本，安装部署我们的overcloud。在这里，又和官方文档有点不一样的，官方文档使用的OpenStackDeploy，我们的operator里面还没有，所以我们需要手动运行。

由于我们的controller是nested kvm，运行起来那是非常慢，整个安装过程在作者的home lab里面要2个多小时。

```bash

# official doc versoin mis-match
# there is no openstackDeploy CRD for current osd operator

# cat << EOF > ${BASE_DIR}/data/install/openstack-deployment.yaml
# apiVersion: osp-director.openstack.org/v1beta1
# kind: OpenStackDeploy
# metadata:
#   name: default
# spec:
#   configVersion: n54dh548h5d7h5f5h648h95h5b5h686h64bhf8h566h65fh5f7h674hdchdh59dh58hf7h667h7h57fh85h557hdh59bh5dh54ch7dh547h579hcfq
#   configGenerator: default
# EOF
# oc create --save-config -f ${BASE_DIR}/data/install/openstack-deployment.yaml -n openstack

oc rsh -n openstack openstackclient

cd /home/cloud-admin
ansible -i /home/cloud-admin/ctlplane-ansible-inventory overcloud -a "sudo dnf -y install python3 lvm2"

# run ansible driven OpenStack deployment
cat << EOF >> /home/cloud-admin/.ssh/config

Host quaylab.infra.redhat.ren
    User root
    IdentityFile /home/cloud-admin/.ssh/git_id_rsa
    StrictHostKeyChecking no

EOF
chmod 600 /home/cloud-admin/.ssh/config

# it is better to run on local machine through crictl exec -it **** bash
./tripleo-deploy.sh -a

/bin/cp tripleo-deploy.sh tripleo-deploy.wzh.sh
sed -i 's/ansible-playbook /ansible-playbook -v /g' tripleo-deploy.wzh.sh
./tripleo-deploy.wzh.sh -p

# because it is nested virtulization
# it will cost almost 2 hours to deploy

# debug 
ssh cloud-admin@192.168.7.43

# podman pull registry.redhat.io/rhosp-rhel8/openstack-cron:16.2
# podman pull registry.redhat.io/rhosp-rhel8/openstack-ovn-controller:16.2
# podman pull registry.redhat.io/rhosp-rhel8/openstack-nova-libvirt:16.2
# podman pull registry.redhat.io/rhosp-rhel8/openstack-iscsid:16.2
# podman pull registry.redhat.io/rhosp-rhel8/openstack-nova-compute:16.2
# podman pull registry.redhat.io/rhosp-rhel8/openstack-neutron-metadata-agent-ovn:16.2


# access the webpage
oc get secret tripleo-passwords -o jsonpath='{.data.tripleo-overcloud-passwords\.yaml}' | base64 -d | grep AdminPassword
  # AdminPassword: 9dhv6qr7xlsbkrzvlvvjtndks
  # CephDashboardAdminPassword: 74rpbjqm8qt586v79rtcwhr2c
  # CephGrafanaAdminPassword: hlg8k7m6fg2zqqvq799xpmsxx
  # HeatStackDomainAdminPassword: flqfdp86gk7xf8rjh2f6nkxhl

# http://172.21.6.40/
# admin / ....
```

![](imgs/2022-11-21-22-09-40.png)

# use the openstack overcloud

我们已经装好了一个openstack overcloud了，接下来，我们就使用一下这个overcloud，创建一个vm试试。

这一步操作，对应到架构图，是这部分：

![](dia/ocp.osp.s99.drawio.svg)

```bash
openstack endpoint list
# +----------------------------------+-----------+--------------+----------------+---------+-----------+-----------------------------------------------+
# | ID                               | Region    | Service Name | Service Type   | Enabled | Interface | URL                                           |
# +----------------------------------+-----------+--------------+----------------+---------+-----------+-----------------------------------------------+
# | 0841809b67a84e8a9f8b5fe1c3fe78b0 | regionOne | glance       | image          | True    | internal  | http://172.17.0.10:9292                       |
# | 0fc84999e6e948cf82d2abb1ff8ffbaf | regionOne | heat         | orchestration  | True    | public    | http://172.21.6.40:8004/v1/%(tenant_id)s      |
# | 1447065d224943c4a3ff886c3bb8c4b3 | regionOne | heat         | orchestration  | True    | internal  | http://172.17.0.10:8004/v1/%(tenant_id)s      |
# | 1c57db71dfcf438b8607cf2549929757 | regionOne | cinderv3     | volumev3       | True    | admin     | http://172.17.0.10:8776/v3/%(tenant_id)s      |
# | 21e24d92d592457782f7c8b39b55ab41 | regionOne | nova         | compute        | True    | public    | http://172.21.6.40:8774/v2.1                  |
# | 26e93f7d318149d492268d7abbeebb8c | regionOne | placement    | placement      | True    | public    | http://172.21.6.40:8778/placement             |
# | 3338679b75b949b0810c807a3dd5b175 | regionOne | heat-cfn     | cloudformation | True    | admin     | http://172.17.0.10:8000/v1                    |
# | 459fa0db22ce4edcba52309bf5157bd6 | regionOne | nova         | compute        | True    | internal  | http://172.17.0.10:8774/v2.1                  |
# | 494e26a5e80d40258e07a32d7f7cd527 | regionOne | placement    | placement      | True    | admin     | http://172.17.0.10:8778/placement             |
# | 4a612665e36840eeb354ebfc1540d372 | regionOne | swift        | object-store   | True    | internal  | http://172.18.0.10:8080/v1/AUTH_%(tenant_id)s |
# | 4ca6bd346e714c86934de3fd80199490 | regionOne | glance       | image          | True    | admin     | http://172.17.0.10:9292                       |
# | 6d1cc6552c4f4bd99b78cfa173f4d30e | regionOne | nova         | compute        | True    | admin     | http://172.17.0.10:8774/v2.1                  |
# | 73a38a88baa342f0882fe846bcf20c23 | regionOne | keystone     | identity       | True    | internal  | http://172.17.0.10:5000                       |
# | 7a194566071140f1a4a88da42e131520 | regionOne | cinderv3     | volumev3       | True    | internal  | http://172.17.0.10:8776/v3/%(tenant_id)s      |
# | 8058f1fd49b6474bab4b2c889bfb8769 | regionOne | cinderv3     | volumev3       | True    | public    | http://172.21.6.40:8776/v3/%(tenant_id)s      |
# | 8158d87f9fb648939e70ebf84398dcb2 | regionOne | neutron      | network        | True    | admin     | http://172.17.0.10:9696                       |
# | 8b6ccf76ea67428ba49431cb58a7d749 | regionOne | cinderv2     | volumev2       | True    | admin     | http://172.17.0.10:8776/v2/%(tenant_id)s      |
# | 8e462b16b50840c380d20a04c05ef19d | regionOne | heat-cfn     | cloudformation | True    | internal  | http://172.17.0.10:8000/v1                    |
# | 96f9cc117a744e74af9a8c889bdcc294 | regionOne | neutron      | network        | True    | internal  | http://172.17.0.10:9696                       |
# | 97996f0256db4ecd97d24dd09d122fed | regionOne | swift        | object-store   | True    | admin     | http://172.18.0.10:8080                       |
# | a874381f22ef480eb858c2062ee0bc84 | regionOne | cinderv2     | volumev2       | True    | internal  | http://172.17.0.10:8776/v2/%(tenant_id)s      |
# | ae9e4589fda4420ebfac1e4d385ebf39 | regionOne | heat-cfn     | cloudformation | True    | public    | http://172.21.6.40:8000/v1                    |
# | c25acb62d7fd4c0c9450c62f99e257e9 | regionOne | neutron      | network        | True    | public    | http://172.21.6.40:9696                       |
# | e16700f441b64798beca1d95982743e0 | regionOne | keystone     | identity       | True    | public    | http://172.21.6.40:5000                       |
# | e433ea6ae0bf4ce19b2fe5424204d35b | regionOne | heat         | orchestration  | True    | admin     | http://172.17.0.10:8004/v1/%(tenant_id)s      |
# | ef151edac10b4838909009e8892fa3a4 | regionOne | placement    | placement      | True    | internal  | http://172.17.0.10:8778/placement             |
# | f31465304ea249f5a4888e52269c6891 | regionOne | keystone     | identity       | True    | admin     | http://192.168.7.40:35357                     |
# | f42b5746d871425db507cf32f4d7d536 | regionOne | cinderv2     | volumev2       | True    | public    | http://172.21.6.40:8776/v2/%(tenant_id)s      |
# | fbe5ee79ad504ef995d111bb2e76a032 | regionOne | swift        | object-store   | True    | public    | http://172.21.6.40:8080/v1/AUTH_%(tenant_id)s |
# | fdefc2aa2c36430294c7c436662cfb16 | regionOne | glance       | image          | True    | public    | http://172.21.6.40:9292                       |
# +----------------------------------+-----------+--------------+----------------+---------+-----------+-----------------------------------------------+


openstack flavor create --ram 512 --disk 1 --vcpu 1 --public tiny
# +----------------------------+--------------------------------------+
# | Field                      | Value                                |
# +----------------------------+--------------------------------------+
# | OS-FLV-DISABLED:disabled   | False                                |
# | OS-FLV-EXT-DATA:ephemeral  | 0                                    |
# | disk                       | 1                                    |
# | id                         | 511a9cc2-9f68-4850-a3e1-de40a68db8d7 |
# | name                       | tiny                                 |
# | os-flavor-access:is_public | True                                 |
# | properties                 |                                      |
# | ram                        | 512                                  |
# | rxtx_factor                | 1.0                                  |
# | swap                       |                                      |
# | vcpus                      | 1                                    |
# +----------------------------+--------------------------------------+

#####################
# on helper
wget https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img

oc cp /data/swap/cirros-0.4.0-x86_64-disk.img openstack/openstackclient:/home/cloud-admin/swap/
# end on helper
#####################

openstack image create cirros --container-format bare --disk-format qcow2 --public --file /home/cloud-admin/swap/cirros-0.4.0-x86_64-disk.img
# +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | Field            | Value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
# +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | checksum         | 443b7623e27ecf03dc9e01ee93f67afe                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
# | container_format | bare                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
# | created_at       | 2022-11-18T15:38:57Z                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
# | disk_format      | qcow2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
# | file             | /v2/images/2d66e2af-8fcb-4d33-8232-5949787a6164/file                                                                                                                                                                                                                                                                                                                                                                                                                                           |
# | id               | 2d66e2af-8fcb-4d33-8232-5949787a6164                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
# | min_disk         | 0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
# | min_ram          | 0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
# | name             | cirros                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
# | owner            | 3647f67bbd5844e38e13f418143c4b57                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
# | properties       | direct_url='rbd://1438a42b-6d15-4143-8e73-8d9f2c9488be/images/2d66e2af-8fcb-4d33-8232-5949787a6164/snap', locations='[{'url': 'rbd://1438a42b-6d15-4143-8e73-8d9f2c9488be/images/2d66e2af-8fcb-4d33-8232-5949787a6164/snap', 'metadata': {'store': 'default_backend'}}]', os_hash_algo='sha512', os_hash_value='6513f21e44aa3da349f248188a44bc304a3653a04122d8fb4535423c8e1d14cd6a153f735bb0982e2161b5b5186106570c17a9e58b64dd39390617cd5a350f78', os_hidden='False', stores='default_backend' |
# | protected        | False                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
# | schema           | /v2/schemas/image                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
# | size             | 12716032                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
# | status           | active                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
# | tags             |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
# | updated_at       | 2022-11-18T15:39:03Z                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
# | virtual_size     | None                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
# | visibility       | public                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
# +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

ssh-keygen -m PEM -t rsa -b 2048 -f ~/.ssh/id_rsa_pem

openstack keypair create --public-key ~/.ssh/id_rsa_pem.pub default
# +-------------+-------------------------------------------------+
# | Field       | Value                                           |
# +-------------+-------------------------------------------------+
# | fingerprint | 64:34:f6:33:9f:87:10:27:d6:5f:80:4c:e7:03:a7:2a |
# | name        | default                                         |
# | user_id     | 0049debaf5d34a83a54486fd418b6981                |
# +-------------+-------------------------------------------------+

openstack security group create basic
# +-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | Field           | Value                                                                                                                                                                     |
# +-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | created_at      | 2022-11-18T15:40:45Z                                                                                                                                                      |
# | description     | basic                                                                                                                                                                     |
# | id              | c3f80883-589f-4cf4-b1b5-059e7966ae82                                                                                                                                      |
# | location        | cloud='overcloud', project.domain_id=, project.domain_name='Default', project.id='3647f67bbd5844e38e13f418143c4b57', project.name='admin', region_name='regionOne', zone= |
# | name            | basic                                                                                                                                                                     |
# | project_id      | 3647f67bbd5844e38e13f418143c4b57                                                                                                                                          |
# | revision_number | 1                                                                                                                                                                         |
# | rules           | created_at='2022-11-18T15:40:46Z', direction='egress', ethertype='IPv4', id='184384ca-4048-4711-9419-b2a9c3c685f8', updated_at='2022-11-18T15:40:46Z'                     |
# |                 | created_at='2022-11-18T15:40:46Z', direction='egress', ethertype='IPv6', id='19cf0c67-6724-49e3-a13c-e366e662b63e', updated_at='2022-11-18T15:40:46Z'                     |
# | tags            | []                                                                                                                                                                        |
# | updated_at      | 2022-11-18T15:40:46Z                                                                                                                                                      |
# +-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

openstack security group rule create basic --protocol tcp --dst-port 22:22 --remote-ip 0.0.0.0/0
# +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | Field             | Value                                                                                                                                                                     |
# +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | created_at        | 2022-11-18T15:41:24Z                                                                                                                                                      |
# | description       |                                                                                                                                                                           |
# | direction         | ingress                                                                                                                                                                   |
# | ether_type        | IPv4                                                                                                                                                                      |
# | id                | a44aa3ee-9fb3-4559-a655-eb90ea974cf8                                                                                                                                      |
# | location          | cloud='overcloud', project.domain_id=, project.domain_name='Default', project.id='3647f67bbd5844e38e13f418143c4b57', project.name='admin', region_name='regionOne', zone= |
# | name              | None                                                                                                                                                                      |
# | port_range_max    | 22                                                                                                                                                                        |
# | port_range_min    | 22                                                                                                                                                                        |
# | project_id        | 3647f67bbd5844e38e13f418143c4b57                                                                                                                                          |
# | protocol          | tcp                                                                                                                                                                       |
# | remote_group_id   | None                                                                                                                                                                      |
# | remote_ip_prefix  | 0.0.0.0/0                                                                                                                                                                 |
# | revision_number   | 0                                                                                                                                                                         |
# | security_group_id | c3f80883-589f-4cf4-b1b5-059e7966ae82                                                                                                                                      |
# | tags              | []                                                                                                                                                                        |
# | updated_at        | 2022-11-18T15:41:24Z                                                                                                                                                      |
# +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

openstack security group rule create --protocol icmp basic
# +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | Field             | Value                                                                                                                                                                     |
# +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | created_at        | 2022-11-18T15:42:26Z                                                                                                                                                      |
# | description       |                                                                                                                                                                           |
# | direction         | ingress                                                                                                                                                                   |
# | ether_type        | IPv4                                                                                                                                                                      |
# | id                | dfe1760d-c76e-4797-9bbe-cf5cbb5a8386                                                                                                                                      |
# | location          | cloud='overcloud', project.domain_id=, project.domain_name='Default', project.id='3647f67bbd5844e38e13f418143c4b57', project.name='admin', region_name='regionOne', zone= |
# | name              | None                                                                                                                                                                      |
# | port_range_max    | None                                                                                                                                                                      |
# | port_range_min    | None                                                                                                                                                                      |
# | project_id        | 3647f67bbd5844e38e13f418143c4b57                                                                                                                                          |
# | protocol          | icmp                                                                                                                                                                      |
# | remote_group_id   | None                                                                                                                                                                      |
# | remote_ip_prefix  | 0.0.0.0/0                                                                                                                                                                 |
# | revision_number   | 0                                                                                                                                                                         |
# | security_group_id | c3f80883-589f-4cf4-b1b5-059e7966ae82                                                                                                                                      |
# | tags              | []                                                                                                                                                                        |
# | updated_at        | 2022-11-18T15:42:26Z                                                                                                                                                      |
# +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

openstack security group rule create --protocol udp --dst-port 53:53 basic
# +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | Field             | Value                                                                                                                                                                     |
# +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | created_at        | 2022-11-18T15:42:58Z                                                                                                                                                      |
# | description       |                                                                                                                                                                           |
# | direction         | ingress                                                                                                                                                                   |
# | ether_type        | IPv4                                                                                                                                                                      |
# | id                | 339c1bfd-e812-472d-84db-77ba47425dfc                                                                                                                                      |
# | location          | cloud='overcloud', project.domain_id=, project.domain_name='Default', project.id='3647f67bbd5844e38e13f418143c4b57', project.name='admin', region_name='regionOne', zone= |
# | name              | None                                                                                                                                                                      |
# | port_range_max    | 53                                                                                                                                                                        |
# | port_range_min    | 53                                                                                                                                                                        |
# | project_id        | 3647f67bbd5844e38e13f418143c4b57                                                                                                                                          |
# | protocol          | udp                                                                                                                                                                       |
# | remote_group_id   | None                                                                                                                                                                      |
# | remote_ip_prefix  | 0.0.0.0/0                                                                                                                                                                 |
# | revision_number   | 0                                                                                                                                                                         |
# | security_group_id | c3f80883-589f-4cf4-b1b5-059e7966ae82                                                                                                                                      |
# | tags              | []                                                                                                                                                                        |
# | updated_at        | 2022-11-18T15:42:58Z                                                                                                                                                      |
# +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

openstack network create --external --provider-physical-network datacentre --provider-network-type flat public
# +---------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | Field                     | Value                                                                                                                                                                     |
# +---------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | admin_state_up            | UP                                                                                                                                                                        |
# | availability_zone_hints   |                                                                                                                                                                           |
# | availability_zones        |                                                                                                                                                                           |
# | created_at                | 2022-11-18T15:47:04Z                                                                                                                                                      |
# | description               |                                                                                                                                                                           |
# | dns_domain                |                                                                                                                                                                           |
# | id                        | 38ae7119-1628-45c6-8763-24dd5eb967cc                                                                                                                                      |
# | ipv4_address_scope        | None                                                                                                                                                                      |
# | ipv6_address_scope        | None                                                                                                                                                                      |
# | is_default                | False                                                                                                                                                                     |
# | is_vlan_transparent       | None                                                                                                                                                                      |
# | location                  | cloud='overcloud', project.domain_id=, project.domain_name='Default', project.id='3647f67bbd5844e38e13f418143c4b57', project.name='admin', region_name='regionOne', zone= |
# | mtu                       | 1500                                                                                                                                                                      |
# | name                      | public                                                                                                                                                                    |
# | port_security_enabled     | True                                                                                                                                                                      |
# | project_id                | 3647f67bbd5844e38e13f418143c4b57                                                                                                                                          |
# | provider:network_type     | flat                                                                                                                                                                      |
# | provider:physical_network | datacentre                                                                                                                                                                |
# | provider:segmentation_id  | None                                                                                                                                                                      |
# | qos_policy_id             | None                                                                                                                                                                      |
# | revision_number           | 1                                                                                                                                                                         |
# | router:external           | External                                                                                                                                                                  |
# | segments                  | None                                                                                                                                                                      |
# | shared                    | False                                                                                                                                                                     |
# | status                    | ACTIVE                                                                                                                                                                    |
# | subnets                   |                                                                                                                                                                           |
# | tags                      |                                                                                                                                                                           |
# | updated_at                | 2022-11-18T15:47:06Z                                                                                                                                                      |
# +---------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

openstack network create --internal private
# +---------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | Field                     | Value                                                                                                                                                                     |
# +---------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | admin_state_up            | UP                                                                                                                                                                        |
# | availability_zone_hints   |                                                                                                                                                                           |
# | availability_zones        |                                                                                                                                                                           |
# | created_at                | 2022-11-18T15:48:08Z                                                                                                                                                      |
# | description               |                                                                                                                                                                           |
# | dns_domain                |                                                                                                                                                                           |
# | id                        | a33927cd-7983-490b-8ab1-e70887abc398                                                                                                                                      |
# | ipv4_address_scope        | None                                                                                                                                                                      |
# | ipv6_address_scope        | None                                                                                                                                                                      |
# | is_default                | False                                                                                                                                                                     |
# | is_vlan_transparent       | None                                                                                                                                                                      |
# | location                  | cloud='overcloud', project.domain_id=, project.domain_name='Default', project.id='3647f67bbd5844e38e13f418143c4b57', project.name='admin', region_name='regionOne', zone= |
# | mtu                       | 1442                                                                                                                                                                      |
# | name                      | private                                                                                                                                                                   |
# | port_security_enabled     | True                                                                                                                                                                      |
# | project_id                | 3647f67bbd5844e38e13f418143c4b57                                                                                                                                          |
# | provider:network_type     | geneve                                                                                                                                                                    |
# | provider:physical_network | None                                                                                                                                                                      |
# | provider:segmentation_id  | 44033                                                                                                                                                                     |
# | qos_policy_id             | None                                                                                                                                                                      |
# | revision_number           | 1                                                                                                                                                                         |
# | router:external           | Internal                                                                                                                                                                  |
# | segments                  | None                                                                                                                                                                      |
# | shared                    | False                                                                                                                                                                     |
# | status                    | ACTIVE                                                                                                                                                                    |
# | subnets                   |                                                                                                                                                                           |
# | tags                      |                                                                                                                                                                           |
# | updated_at                | 2022-11-18T15:48:08Z                                                                                                                                                      |
# +---------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


export GATEWAY=172.21.6.254
# export STANDALONE_HOST=192.168.25.2
export PUBLIC_NETWORK_CIDR=172.21.6.0/24
export PRIVATE_NETWORK_CIDR=192.168.100.0/24
export PUBLIC_NET_START=172.21.6.70
export PUBLIC_NET_END=172.21.6.80
export DNS_SERVER=172.21.1.1

openstack subnet create public-net \
    --subnet-range $PUBLIC_NETWORK_CIDR \
    --no-dhcp \
    --gateway $GATEWAY \
    --allocation-pool start=$PUBLIC_NET_START,end=$PUBLIC_NET_END \
    --network public
# +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | Field             | Value                                                                                                                                                                     |
# +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | allocation_pools  | 172.21.6.70-172.21.6.80                                                                                                                                                   |
# | cidr              | 172.21.6.0/24                                                                                                                                                             |
# | created_at        | 2022-11-18T15:51:01Z                                                                                                                                                      |
# | description       |                                                                                                                                                                           |
# | dns_nameservers   |                                                                                                                                                                           |
# | enable_dhcp       | False                                                                                                                                                                     |
# | gateway_ip        | 172.21.6.254                                                                                                                                                              |
# | host_routes       |                                                                                                                                                                           |
# | id                | 812aa93f-aa5b-42b5-97ef-63ae59e8c1da                                                                                                                                      |
# | ip_version        | 4                                                                                                                                                                         |
# | ipv6_address_mode | None                                                                                                                                                                      |
# | ipv6_ra_mode      | None                                                                                                                                                                      |
# | location          | cloud='overcloud', project.domain_id=, project.domain_name='Default', project.id='3647f67bbd5844e38e13f418143c4b57', project.name='admin', region_name='regionOne', zone= |
# | name              | public-net                                                                                                                                                                |
# | network_id        | 38ae7119-1628-45c6-8763-24dd5eb967cc                                                                                                                                      |
# | prefix_length     | None                                                                                                                                                                      |
# | project_id        | 3647f67bbd5844e38e13f418143c4b57                                                                                                                                          |
# | revision_number   | 0                                                                                                                                                                         |
# | segment_id        | None                                                                                                                                                                      |
# | service_types     |                                                                                                                                                                           |
# | subnetpool_id     | None                                                                                                                                                                      |
# | tags              |                                                                                                                                                                           |
# | updated_at        | 2022-11-18T15:51:01Z                                                                                                                                                      |
# +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

openstack subnet create private-net \
    --subnet-range $PRIVATE_NETWORK_CIDR \
    --network private
# +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | Field             | Value                                                                                                                                                                     |
# +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | allocation_pools  | 192.168.100.2-192.168.100.254                                                                                                                                             |
# | cidr              | 192.168.100.0/24                                                                                                                                                          |
# | created_at        | 2022-11-18T15:52:06Z                                                                                                                                                      |
# | description       |                                                                                                                                                                           |
# | dns_nameservers   |                                                                                                                                                                           |
# | enable_dhcp       | True                                                                                                                                                                      |
# | gateway_ip        | 192.168.100.1                                                                                                                                                             |
# | host_routes       |                                                                                                                                                                           |
# | id                | 0a378d92-386f-437b-acf8-564595e394ba                                                                                                                                      |
# | ip_version        | 4                                                                                                                                                                         |
# | ipv6_address_mode | None                                                                                                                                                                      |
# | ipv6_ra_mode      | None                                                                                                                                                                      |
# | location          | cloud='overcloud', project.domain_id=, project.domain_name='Default', project.id='3647f67bbd5844e38e13f418143c4b57', project.name='admin', region_name='regionOne', zone= |
# | name              | private-net                                                                                                                                                               |
# | network_id        | a33927cd-7983-490b-8ab1-e70887abc398                                                                                                                                      |
# | prefix_length     | None                                                                                                                                                                      |
# | project_id        | 3647f67bbd5844e38e13f418143c4b57                                                                                                                                          |
# | revision_number   | 0                                                                                                                                                                         |
# | segment_id        | None                                                                                                                                                                      |
# | service_types     |                                                                                                                                                                           |
# | subnetpool_id     | None                                                                                                                                                                      |
# | tags              |                                                                                                                                                                           |
# | updated_at        | 2022-11-18T15:52:06Z                                                                                                                                                      |
# +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

# NOTE: In this case an IP will be automatically assigned
# from the allocation pool for the subnet.
openstack router create vrouter
# +-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | Field                   | Value                                                                                                                                                                     |
# +-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | admin_state_up          | UP                                                                                                                                                                        |
# | availability_zone_hints |                                                                                                                                                                           |
# | availability_zones      |                                                                                                                                                                           |
# | created_at              | 2022-11-18T15:53:12Z                                                                                                                                                      |
# | description             |                                                                                                                                                                           |
# | external_gateway_info   | null                                                                                                                                                                      |
# | flavor_id               | None                                                                                                                                                                      |
# | id                      | cb1fcb45-1716-4676-8ecd-9a0ee22ce936                                                                                                                                      |
# | location                | cloud='overcloud', project.domain_id=, project.domain_name='Default', project.id='3647f67bbd5844e38e13f418143c4b57', project.name='admin', region_name='regionOne', zone= |
# | name                    | vrouter                                                                                                                                                                   |
# | project_id              | 3647f67bbd5844e38e13f418143c4b57                                                                                                                                          |
# | revision_number         | 1                                                                                                                                                                         |
# | routes                  |                                                                                                                                                                           |
# | status                  | ACTIVE                                                                                                                                                                    |
# | tags                    |                                                                                                                                                                           |
# | updated_at              | 2022-11-18T15:53:12Z                                                                                                                                                      |
# +-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

openstack router set vrouter --external-gateway public

openstack router add subnet vrouter private-net

openstack floating ip create public
# +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | Field               | Value                                                                                                                                                                                               |
# +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# | created_at          | 2022-11-18T15:56:17Z                                                                                                                                                                                |
# | description         |                                                                                                                                                                                                     |
# | dns_domain          |                                                                                                                                                                                                     |
# | dns_name            |                                                                                                                                                                                                     |
# | fixed_ip_address    | None                                                                                                                                                                                                |
# | floating_ip_address | 172.21.6.79                                                                                                                                                                                         |
# | floating_network_id | 38ae7119-1628-45c6-8763-24dd5eb967cc                                                                                                                                                                |
# | id                  | de30c4aa-3aac-4216-af83-f335aac2765e                                                                                                                                                                |
# | location            | Munch({'cloud': 'overcloud', 'region_name': 'regionOne', 'zone': None, 'project': Munch({'id': '3647f67bbd5844e38e13f418143c4b57', 'name': 'admin', 'domain_id': None, 'domain_name': 'Default'})}) |
# | name                | 172.21.6.79                                                                                                                                                                                         |
# | port_details        | None                                                                                                                                                                                                |
# | port_id             | None                                                                                                                                                                                                |
# | project_id          | 3647f67bbd5844e38e13f418143c4b57                                                                                                                                                                    |
# | qos_policy_id       | None                                                                                                                                                                                                |
# | revision_number     | 0                                                                                                                                                                                                   |
# | router_id           | None                                                                                                                                                                                                |
# | status              | DOWN                                                                                                                                                                                                |
# | subnet_id           | None                                                                                                                                                                                                |
# | tags                | []                                                                                                                                                                                                  |
# | updated_at          | 2022-11-18T15:56:17Z                                                                                                                                                                                |
# +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


openstack server create --flavor tiny --image cirros --key-name default --network private --security-group basic myserver
# +-------------------------------------+-----------------------------------------------+
# | Field                               | Value                                         |
# +-------------------------------------+-----------------------------------------------+
# | OS-DCF:diskConfig                   | MANUAL                                        |
# | OS-EXT-AZ:availability_zone         |                                               |
# | OS-EXT-SRV-ATTR:host                | None                                          |
# | OS-EXT-SRV-ATTR:hypervisor_hostname | None                                          |
# | OS-EXT-SRV-ATTR:instance_name       |                                               |
# | OS-EXT-STS:power_state              | NOSTATE                                       |
# | OS-EXT-STS:task_state               | scheduling                                    |
# | OS-EXT-STS:vm_state                 | building                                      |
# | OS-SRV-USG:launched_at              | None                                          |
# | OS-SRV-USG:terminated_at            | None                                          |
# | accessIPv4                          |                                               |
# | accessIPv6                          |                                               |
# | addresses                           |                                               |
# | adminPass                           | r9QVNEs5r8Ji                                  |
# | config_drive                        |                                               |
# | created                             | 2022-11-18T15:57:49Z                          |
# | flavor                              | tiny (511a9cc2-9f68-4850-a3e1-de40a68db8d7)   |
# | hostId                              |                                               |
# | id                                  | c6488d98-bdc8-4439-a586-f74c7d31e64d          |
# | image                               | cirros (2d66e2af-8fcb-4d33-8232-5949787a6164) |
# | key_name                            | default                                       |
# | name                                | myserver                                      |
# | progress                            | 0                                             |
# | project_id                          | 3647f67bbd5844e38e13f418143c4b57              |
# | properties                          |                                               |
# | security_groups                     | name='c3f80883-589f-4cf4-b1b5-059e7966ae82'   |
# | status                              | BUILD                                         |
# | updated                             | 2022-11-18T15:57:50Z                          |
# | user_id                             | 0049debaf5d34a83a54486fd418b6981              |
# | volumes_attached                    |                                               |
# +-------------------------------------+-----------------------------------------------+

openstack server add floating ip myserver 172.21.6.79

ssh -i ~/.ssh/id_rsa_pem cirros@172.21.6.79
ip a
# 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
#     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
#     inet 127.0.0.1/8 scope host lo
#        valid_lft forever preferred_lft forever
#     inet6 ::1/128 scope host
#        valid_lft forever preferred_lft forever
# 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc pfifo_fast qlen 1000
#     link/ether fa:16:3e:f5:62:0f brd ff:ff:ff:ff:ff:ff
#     inet 192.168.100.108/24 brd 192.168.100.255 scope global eth0
#        valid_lft forever preferred_lft forever
#     inet6 fe80::f816:3eff:fef5:620f/64 scope link
#        valid_lft forever preferred_lft forever
```

## network config on computerHCI

```
[root@computehci-0 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:4a:71:e9 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e112:e339:d036:a9ae/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:ec:40:fa brd ff:ff:ff:ff:ff:ff
    inet 172.22.0.25/24 brd 172.22.0.255 scope global dynamic noprefixroute enp2s0
       valid_lft 79sec preferred_lft 79sec
    inet6 fe80::a494:89f6:d082:30ec/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
    link/ether 52:54:00:7c:23:1a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe7c:231a/64 scope link
       valid_lft forever preferred_lft forever
5: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:36:ee:42 brd ff:ff:ff:ff:ff:ff
    inet 192.168.7.43/24 brd 192.168.7.255 scope global enp4s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe36:ee42/64 scope link
       valid_lft forever preferred_lft forever
6: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:88:1f:84 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::67c4:3ad8:90c6:9196/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
7: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:92:b5:ed brd ff:ff:ff:ff:ff:ff
    inet6 fe80::6f9:254d:87cb:7f3e/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
8: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:d9:66:89 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::d41b:2e70:f90:39a7/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
9: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:d9:61:82 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b779:5bdb:b449:3d9/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
10: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 16:c6:40:67:40:f9 brd ff:ff:ff:ff:ff:ff
11: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:54:00:7c:23:1a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe7c:231a/64 scope link
       valid_lft forever preferred_lft forever
12: vlan50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ea:13:bc:e1:ae:8d brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.11/24 brd 172.20.0.255 scope global vlan50
       valid_lft forever preferred_lft forever
    inet6 fe80::e813:bcff:fee1:ae8d/64 scope link
       valid_lft forever preferred_lft forever
13: vlan30@enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:36:ee:42 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.12/24 brd 172.18.0.255 scope global vlan30
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe36:ee42/64 scope link
       valid_lft forever preferred_lft forever
14: vlan40@enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:36:ee:42 brd ff:ff:ff:ff:ff:ff
    inet 172.19.0.12/24 brd 172.19.0.255 scope global vlan40
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe36:ee42/64 scope link
       valid_lft forever preferred_lft forever
15: vlan20@enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:36:ee:42 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.13/24 brd 172.17.0.255 scope global vlan20
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe36:ee42/64 scope link
       valid_lft forever preferred_lft forever
16: br-int: <BROADCAST,MULTICAST> mtu 1442 qdisc noop state DOWN group default qlen 1000
    link/ether 3a:0d:9a:c5:a4:eb brd ff:ff:ff:ff:ff:ff
17: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 92:b3:8a:d2:73:ac brd ff:ff:ff:ff:ff:ff
    inet6 fe80::90b3:8aff:fed2:73ac/64 scope link
       valid_lft forever preferred_lft forever
18: tapa46cc8cc-20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether fe:16:3e:f5:62:0f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fef5:620f/64 scope link
       valid_lft forever preferred_lft forever
19: tapa33927cd-70@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 6a:49:ac:e8:bd:37 brd ff:ff:ff:ff:ff:ff link-netns ovnmeta-a33927cd-7983-490b-8ab1-e70887abc398
    inet6 fe80::6849:acff:fee8:bd37/64 scope link
       valid_lft forever preferred_lft forever

[root@computehci-0 ~]# ip netns exec  ovnmeta-a33927cd-7983-490b-8ab1-e70887abc398 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: tapa33927cd-71@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fa:16:3e:b0:e5:a8 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.169.254/16 brd 169.254.255.255 scope global tapa33927cd-71
       valid_lft forever preferred_lft forever
    inet 192.168.100.2/24 brd 192.168.100.255 scope global tapa33927cd-71
       valid_lft forever preferred_lft forever

[root@computehci-0 ~]# ovs-vsctl show
54ff9cc9-ffe9-4a1d-9136-4a15c60f43dd
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-ex
        fail_mode: standalone
        Port patch-provnet-bfad145a-fb08-420c-8454-2c69a86ac674-to-br-int
            Interface patch-provnet-bfad145a-fb08-420c-8454-2c69a86ac674-to-br-int
                type: patch
                options: {peer=patch-br-int-to-provnet-bfad145a-fb08-420c-8454-2c69a86ac674}
        Port vlan50
            tag: 50
            Interface vlan50
                type: internal
        Port br-ex
            Interface br-ex
                type: internal
        Port enp3s0
            Interface enp3s0
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port patch-br-int-to-provnet-bfad145a-fb08-420c-8454-2c69a86ac674
            Interface patch-br-int-to-provnet-bfad145a-fb08-420c-8454-2c69a86ac674
                type: patch
                options: {peer=patch-provnet-bfad145a-fb08-420c-8454-2c69a86ac674-to-br-int}
        Port tapa46cc8cc-20
            Interface tapa46cc8cc-20
        Port ovn-8c68c1-0
            Interface ovn-8c68c1-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="172.20.0.10"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}
        Port br-int
            Interface br-int
                type: internal
        Port tapa33927cd-70
            Interface tapa33927cd-70
    ovs_version: "2.15.7"

[root@computehci-0 ~]# ovs-dpctl dump-flows | grep 172.21.6.79
recirc_id(0),in_port(4),eth(src=00:0c:29:d9:f0:d9,dst=fa:16:3e:df:80:af),eth_type(0x0800),ipv4(src=172.21.6.0/255.255.255.192,dst=172.21.6.79,proto=1,ttl=64,frag=no), packets:1962, bytes:192276, used:0.645s, actions:ct(zone=1,nat),recirc(0x3)


[root@computehci-0 ~]# ovs-dpctl dump-flows
recirc_id(0),in_port(4),eth(src=02:2e:8d:00:00:06,dst=32:df:00:f0:a3:4d),eth_type(0x8100),vlan(vid=50,pcp=0),encap(eth_type(0x0800),ipv4(frag=no)), packets:3438, bytes:412560, used:0.467s, actions:pop_vlan,5
recirc_id(0),in_port(4),eth(src=52:54:00:d9:66:89,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(src=0.0.0.0/255.0.0.0,dst=240.0.0.0/240.0.0.0,ttl=64,frag=no), packets:0, bytes:0, used:never, actions:3
recirc_id(0),in_port(4),eth(src=00:0c:29:d9:f0:d9,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=192.168.7.11,tip=192.168.7.22,op=1/0xff), packets:2111, bytes:126660, used:0.018s, actions:3
recirc_id(0),in_port(4),eth(src=90:b1:1c:44:d6:0f,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=172.21.6.103,tip=172.21.6.102,op=1/0xff,sha=90:b1:1c:44:d6:0f), packets:2280, bytes:95760, used:1.595s, actions:3
recirc_id(0x3),in_port(4),ct_state(-new+est-rel-rpl-inv+trk),ct_mark(0/0x1),eth(src=00:0c:29:d9:f0:d9,dst=fa:16:3e:df:80:af),eth_type(0x0800),ipv4(dst=192.168.100.50,proto=1,ttl=64,frag=no), packets:1988, bytes:194824, used:0.580s, actions:ct_clear,set(eth(src=fa:16:3e:89:22:a5,dst=fa:16:3e:bd:98:6b)),set(ipv4(ttl=63)),ct(zone=7),recirc(0x6)
recirc_id(0),in_port(6),eth(src=fa:16:3e:bd:98:6b,dst=fa:16:3e:89:22:a5),eth_type(0x0800),ipv4(src=192.168.100.50,dst=172.21.6.0/255.255.255.192,proto=1,frag=no), packets:1989, bytes:194922, used:0.579s, actions:ct(zone=7),recirc(0xa)
recirc_id(0),in_port(4),eth(src=52:54:00:3d:23:56,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(src=0.0.0.0/255.0.0.0,dst=240.0.0.0/240.0.0.0,ttl=64,frag=no), packets:0, bytes:0, used:never, actions:3
recirc_id(0),in_port(4),eth(src=52:54:00:aa:41:08,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=192.168.7.101,tip=192.168.7.101,op=1/0xff), packets:0, bytes:0, used:never, actions:3
recirc_id(0),in_port(4),eth(src=52:54:00:1e:bb:a4,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(src=0.0.0.0/255.0.0.0,dst=240.0.0.0/240.0.0.0,ttl=64,frag=no), packets:0, bytes:0, used:never, actions:3
recirc_id(0),tunnel(tun_id=0x0,src=172.20.0.10,dst=172.20.0.11,flags(-df+csum+key)),in_port(2),eth(),eth_type(0x0800),ipv4(proto=17,frag=no),udp(dst=3784), packets:3437, bytes:226842, used:0.467s, actions:userspace(pid=3165666405,slow_path(bfd))
recirc_id(0),in_port(5),eth(src=32:df:00:f0:a3:4d,dst=02:2e:8d:00:00:06),eth_type(0x0800),ipv4(frag=no), packets:3458, bytes:401128, used:0.744s, actions:push_vlan(vid=50,pcp=0),4
recirc_id(0),in_port(4),eth(src=52:54:00:d9:66:89,dst=33:33:00:00:00:16),eth_type(0x86dd),ipv6(dst=ff02::16,proto=58,hlimit=1,frag=no), packets:1, bytes:90, used:5.509s, actions:3
recirc_id(0xe),in_port(6),ct_state(-new+est-rel+rpl-inv+trk),ct_mark(0/0x1),eth(src=fa:16:3e:df:80:af,dst=00:0c:29:d9:f0:d9),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=172.21.6.0/255.255.255.192,frag=no), packets:1988, bytes:194824, used:0.580s, actions:ct_clear,4
recirc_id(0),in_port(4),eth(src=00:17:94:73:12:8b,dst=01:00:0c:cc:cc:cc),eth_type(0/0xffff), packets:1, bytes:398, used:7.440s, actions:drop
recirc_id(0),in_port(4),eth(src=52:54:00:95:3f:da,dst=33:33:00:00:00:16),eth_type(0x86dd),ipv6(dst=ff02::16,proto=58,hlimit=1,frag=no), packets:1, bytes:150, used:3.723s, actions:3
recirc_id(0),in_port(4),eth(src=52:54:00:95:3f:da,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(src=0.0.0.0/255.0.0.0,dst=240.0.0.0/240.0.0.0,ttl=64,frag=no), packets:0, bytes:0, used:never, actions:3
recirc_id(0),in_port(4),eth(src=02:2e:8d:00:00:06,dst=32:df:00:f0:a3:4d),eth_type(0x8100),vlan(vid=50,pcp=0),encap(eth_type(0x0806)), packets:1, bytes:46, used:5.278s, actions:pop_vlan,5
recirc_id(0xa),in_port(6),ct_state(-new+est-rel+rpl-inv+trk),ct_mark(0/0x1),eth(src=fa:16:3e:bd:98:6b,dst=fa:16:3e:89:22:a5),eth_type(0x0800),ipv4(src=192.168.100.50,dst=172.21.6.11,proto=1,ttl=64,frag=no), packets:1989, bytes:194922, used:0.580s, actions:ct_clear,set(eth(src=fa:16:3e:df:80:af,dst=00:0c:29:d9:f0:d9)),set(ipv4(ttl=63)),ct(zone=1,nat),recirc(0xe)
recirc_id(0),in_port(4),eth(src=fe:54:00:7c:23:1a,dst=01:80:c2:00:00:00),eth_type(0/0xffff), packets:15467, bytes:804284, used:0.468s, actions:drop
recirc_id(0),in_port(4),eth(src=00:0c:29:d9:f0:d9,dst=fa:16:3e:df:80:af),eth_type(0x0800),ipv4(src=172.21.6.0/255.255.255.192,dst=172.21.6.79,proto=1,ttl=64,frag=no), packets:2011, bytes:197078, used:0.581s, actions:ct(zone=1,nat),recirc(0x3)
recirc_id(0x6),in_port(4),ct_state(-new+est-rel-rpl-inv+trk),ct_mark(0/0x1),eth(src=fa:16:3e:89:22:a5,dst=fa:16:3e:bd:98:6b),eth_type(0x0800),ipv4(dst=192.168.100.50,proto=1,frag=no), packets:1988, bytes:194824, used:0.581s, actions:6
recirc_id(0),in_port(4),eth(src=52:54:00:73:0f:fb,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(src=0.0.0.0/255.0.0.0,dst=240.0.0.0/240.0.0.0,ttl=64,frag=no), packets:0, bytes:0, used:never, actions:3
recirc_id(0),in_port(4),eth(src=52:54:00:3d:23:56,dst=33:33:00:00:00:16),eth_type(0x86dd),ipv6(dst=ff02::16,proto=58,hlimit=1,frag=no), packets:1, bytes:150, used:1.293s, actions:3
recirc_id(0),in_port(5),eth(src=32:df:00:f0:a3:4d,dst=02:2e:8d:00:00:06),eth_type(0x0806), packets:1, bytes:42, used:5.278s, actions:push_vlan(vid=50,pcp=0),4
recirc_id(0),in_port(4),eth(src=52:54:00:73:0f:fb,dst=33:33:00:00:00:16),eth_type(0x86dd),ipv6(dst=ff02::16,proto=58,hlimit=1,frag=no), packets:1, bytes:150, used:9.113s, actions:3
recirc_id(0),in_port(4),eth(src=00:0c:29:d9:f0:d9,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=192.168.7.11,tip=192.168.7.26,op=1/0xff), packets:2117, bytes:127020, used:0.021s, actions:3
recirc_id(0),in_port(4),eth(src=52:54:00:1e:bb:a4,dst=33:33:00:00:00:16),eth_type(0x86dd),ipv6(dst=ff02::16,proto=58,hlimit=1,frag=no), packets:1, bytes:150, used:5.710s, actions:3

```

## network config on contoller

```
[root@controller-0 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc mq state UP group default qlen 1000
    link/ether 02:2e:8d:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.2/24 brd 10.0.2.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::2e:8dff:fe00:0/64 scope link
       valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 02:2e:8d:00:00:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.7.42/24 brd 192.168.7.255 scope global enp2s0
       valid_lft forever preferred_lft forever
    inet 192.168.7.40/32 brd 192.168.7.255 scope global enp2s0
       valid_lft forever preferred_lft forever
    inet6 fe80::2e:8dff:fe00:1/64 scope link
       valid_lft forever preferred_lft forever
4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 02:2e:8d:00:00:02 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2e:8dff:fe00:2/64 scope link
       valid_lft forever preferred_lft forever
5: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc mq state UP group default qlen 1000
    link/ether 02:2e:8d:00:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.12/24 brd 172.17.0.255 scope global enp4s0
       valid_lft forever preferred_lft forever
    inet 172.17.0.10/32 brd 172.17.0.255 scope global enp4s0
       valid_lft forever preferred_lft forever
    inet6 fe80::2e:8dff:fe00:3/64 scope link
       valid_lft forever preferred_lft forever
6: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 02:2e:8d:00:00:04 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.11/24 brd 172.18.0.255 scope global enp5s0
       valid_lft forever preferred_lft forever
    inet 172.18.0.10/32 brd 172.18.0.255 scope global enp5s0
       valid_lft forever preferred_lft forever
    inet6 fe80::2e:8dff:fe00:4/64 scope link
       valid_lft forever preferred_lft forever
7: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 02:2e:8d:00:00:05 brd ff:ff:ff:ff:ff:ff
    inet 172.19.0.11/24 brd 172.19.0.255 scope global enp6s0
       valid_lft forever preferred_lft forever
    inet 172.19.0.10/32 brd 172.19.0.255 scope global enp6s0
       valid_lft forever preferred_lft forever
    inet6 fe80::2e:8dff:fe00:5/64 scope link
       valid_lft forever preferred_lft forever
8: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 02:2e:8d:00:00:06 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2e:8dff:fe00:6/64 scope link
       valid_lft forever preferred_lft forever
9: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7a:50:65:fe:ff:25 brd ff:ff:ff:ff:ff:ff
10: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 02:2e:8d:00:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.21.6.42/24 brd 172.21.6.255 scope global br-ex
       valid_lft forever preferred_lft forever
    inet 172.21.6.40/32 brd 172.21.6.255 scope global br-ex
       valid_lft forever preferred_lft forever
    inet6 fe80::2e:8dff:fe00:2/64 scope link
       valid_lft forever preferred_lft forever
11: br-tenant: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 02:2e:8d:00:00:06 brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.10/24 brd 172.20.0.255 scope global br-tenant
       valid_lft forever preferred_lft forever
    inet6 fe80::2e:8dff:fe00:6/64 scope link
       valid_lft forever preferred_lft forever
12: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7a:5c:8b:47:78:45 brd ff:ff:ff:ff:ff:ff
13: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether ce:f7:9c:bb:43:b7 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ccf7:9cff:febb:43b7/64 scope link
       valid_lft forever preferred_lft forever



[root@controller-0 ~]# ovs-vsctl show
78811126-7766-4789-98e3-e8c3cf1cff0e
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port patch-br-int-to-provnet-bfad145a-fb08-420c-8454-2c69a86ac674
            Interface patch-br-int-to-provnet-bfad145a-fb08-420c-8454-2c69a86ac674
                type: patch
                options: {peer=patch-provnet-bfad145a-fb08-420c-8454-2c69a86ac674-to-br-int}
        Port ovn-980774-0
            Interface ovn-980774-0
                type: geneve
                options: {csum="true", key=flow, remote_ip="172.20.0.11"}
                bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}
        Port br-int
            Interface br-int
                type: internal
    Bridge br-tenant
        fail_mode: standalone
        Port enp7s0
            Interface enp7s0
        Port br-tenant
            Interface br-tenant
                type: internal
    Bridge br-ex
        fail_mode: standalone
        Port br-ex
            Interface br-ex
                type: internal
        Port patch-provnet-bfad145a-fb08-420c-8454-2c69a86ac674-to-br-int
            Interface patch-provnet-bfad145a-fb08-420c-8454-2c69a86ac674-to-br-int
                type: patch
                options: {peer=patch-br-int-to-provnet-bfad145a-fb08-420c-8454-2c69a86ac674}
        Port enp3s0
            Interface enp3s0
    ovs_version: "2.15.7"


[root@controller-0 ~]# ovs-dpctl dump-flows
recirc_id(0),in_port(2),eth(src=00:0c:29:d9:f0:d9,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=192.168.7.11,tip=192.168.7.26,op=1/0xff), packets:2057, bytes:123420, used:0.758s, actions:1
recirc_id(0),in_port(2),eth(src=00:17:94:73:12:8b,dst=01:00:0c:00:00:00),eth_type(0/0xffff), packets:0, bytes:0, used:never, actions:drop
recirc_id(0),in_port(5),eth(src=00:17:94:73:12:8b,dst=01:00:0c:00:00:00),eth_type(0/0xffff), packets:0, bytes:0, used:never, actions:drop
recirc_id(0),in_port(5),eth(src=52:54:00:4a:71:e9,dst=33:33:00:00:00:16),eth_type(0x86dd),ipv6(frag=no), packets:1, bytes:90, used:9.340s, actions:6
recirc_id(0),in_port(5),eth(src=00:0c:29:d9:f0:d9,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=192.168.7.11,tip=192.168.7.22,op=1/0xff), packets:2046, bytes:122760, used:0.756s, actions:6
recirc_id(0),in_port(2),eth(src=00:17:94:73:12:8b,dst=01:00:0c:cc:cc:cc),eth_type(0/0xffff), packets:0, bytes:0, used:never, actions:drop
recirc_id(0),in_port(5),eth(src=fe:54:00:b9:09:0e,dst=01:80:c2:00:00:00),eth_type(0/0xffff), packets:1706, bytes:88712, used:0.181s, actions:drop
recirc_id(0),in_port(6),eth(src=02:2e:8d:00:00:06,dst=32:df:00:f0:a3:4d),eth_type(0x0806), packets:1, bytes:42, used:0.636s, actions:5
recirc_id(0),tunnel(tun_id=0x0,src=172.20.0.11,dst=172.20.0.10,flags(-df+csum+key)),in_port(4),eth(),eth_type(0x0800),ipv4(proto=17,frag=no),udp(dst=3784), packets:3544, bytes:233904, used:0.822s, actions:userspace(pid=2361299521,slow_path(bfd))
recirc_id(0),in_port(2),eth(src=00:0c:29:d9:f0:d9,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=192.168.7.11,tip=192.168.7.22,op=1/0xff), packets:2057, bytes:123420, used:0.758s, actions:1
recirc_id(0),in_port(5),eth(src=32:df:00:f0:a3:4d,dst=02:2e:8d:00:00:06),eth_type(0x0806), packets:1, bytes:42, used:0.636s, actions:6
recirc_id(0),in_port(5),eth(src=00:17:94:73:12:8b,dst=01:00:0c:cc:cc:cc),eth_type(0/0xffff), packets:0, bytes:0, used:never, actions:drop
recirc_id(0),in_port(2),eth(src=fe:54:00:5c:61:1f,dst=01:80:c2:00:00:00),eth_type(0/0xffff), packets:1714, bytes:89128, used:0.181s, actions:drop
recirc_id(0),in_port(2),eth(src=52:54:00:72:47:a9,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=172.22.0.6,tip=172.22.0.240,op=1/0xff), packets:0, bytes:0, used:never, actions:1
recirc_id(0),in_port(5),eth(src=52:54:00:72:47:a9,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=172.22.0.6,tip=172.22.0.240,op=1/0xff), packets:0, bytes:0, used:never, actions:6
recirc_id(0),in_port(2),eth(src=90:b1:1c:44:d6:0f,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=172.21.6.103,tip=172.21.6.102,op=1/0xff), packets:2584, bytes:108528, used:1.372s, actions:1
recirc_id(0),in_port(5),eth(src=32:df:00:f0:a3:4d,dst=02:2e:8d:00:00:06),eth_type(0x0800),ipv4(frag=no), packets:3545, bytes:411220, used:0.822s, actions:6
recirc_id(0),in_port(5),eth(src=90:b1:1c:44:d6:0f,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=172.21.6.103,tip=172.21.6.102,op=1/0xff), packets:2571, bytes:107982, used:1.371s, actions:6
recirc_id(0),in_port(5),eth(src=00:0c:29:d9:f0:d9,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=192.168.7.11,tip=192.168.7.26,op=1/0xff), packets:2046, bytes:122760, used:0.756s, actions:6
recirc_id(0),in_port(6),eth(src=02:2e:8d:00:00:06,dst=32:df:00:f0:a3:4d),eth_type(0x0800),ipv4(frag=no), packets:3525, bytes:408900, used:0.280s, actions:5
recirc_id(0),in_port(2),eth(src=52:54:00:4a:71:e9,dst=33:33:00:00:00:16),eth_type(0x86dd),ipv6(src=fe80::/ffc0::,dst=ff02::16,proto=58,hlimit=1,frag=no),icmpv6(type=143), packets:1, bytes:90, used:9.344s, actions:1

```

# tips

## delete 

```bash

oc delete -f ${BASE_DIR}/data/install/overcloud-network.yaml -n openstack
oc delete -f ${BASE_DIR}/data/install/ctlplane-network.yaml -n openstack
oc delete -f ${BASE_DIR}/data/install/openstack-userpassword.yaml -n openstack
oc delete -f ${BASE_DIR}/data/install/osp-director-operator.yaml

oc delete project openstack

```

## fix nics

```bash
virsh list --all
#  Id   Name                     State
# -----------------------------------------
#  -    ocp4-acm-hub-master01    shut off
#  -    ocp4-acm-one-bootstrap   shut off
#  -    ocp4-acm-one-master-01   shut off
#  -    ocp4-acm-one-master-02   shut off
#  -    ocp4-acm-one-master-03   shut off
#  -    ocp4-ipi-osp-master-01   shut off
#  -    ocp4-ipi-osp-master-02   shut off
#  -    ocp4-ipi-osp-master-03   shut off
#  -    ocp4-ipi-osp-worker-01   shut off
#  -    osp-17-0-all-in-one      shut off

for i in {1..3}
do
  for j in {1..4}
  do
    echo ocp4-ipi-osp-master-0$i
    # virsh attach-interface --domain ocp4-ipi-osp-master-0$i --type bridge --source baremetal --model virtio
  done
done

for i in 23 24 25
do
  ssh root@192.168.7.$i "nmcli con del br-osp "
done


```

# question

1. network topo
2. 2 host with ovs and vlan

# end