|
| 1 | +--- |
| 2 | +title: Arbiter |
| 3 | +--- |
| 4 | + |
| 5 | +# Arbiter |
| 6 | + |
| 7 | +## About This Project |
| 8 | + |
| 9 | +The external-arbiter-operator (Arbiter) works with Rook-provisioned Ceph |
| 10 | +clusters to deploy external arbiters (monitors) that are not managed by Rook |
| 11 | +but that participate in consensus. |
| 12 | + |
| 13 | +The operator also monitors the remote cluster to verify its availability and |
| 14 | +ensure that the tenant has sufficient permissions to handle the deployment of |
| 15 | +Arbiter. |
| 16 | + |
| 17 | +## Requirements and Setup |
| 18 | + |
| 19 | +### Required Tools |
| 20 | + |
| 21 | +The following tools are required on your development machine: |
| 22 | + |
| 23 | +- `sed` |
| 24 | +- `openssl` |
| 25 | +- `make` |
| 26 | +- `git` |
| 27 | +- `golang` |
| 28 | +- `lima` (or another method to provision Kubernetes locally, such as Minikube) |
| 29 | +- `kubectl` |
| 30 | +- `docker` (or any compatible container engine, such as Podman) |
| 31 | +- `helm` |
| 32 | + |
| 33 | +The remaining dependencies are provisioned via Go tools, including the |
| 34 | +Kubebuilder toolset. |
| 35 | + |
| 36 | +## Quick Start |
| 37 | + |
| 38 | +What follows is a quick walkthrough on how to prepare the environment, run the |
| 39 | +operator locally, and deploy an external monitor. |
| 40 | + |
| 41 | +### Clone and Setup |
| 42 | + |
| 43 | +```bash |
| 44 | +# Clone the Rook repository: https://github.com/rook/rook |
| 45 | + |
| 46 | +#Run `make deps`: |
| 47 | +make deps |
| 48 | + |
| 49 | +# Create OSD for Ceph |
| 50 | +limactl disk create osd --size=8G |
| 51 | + |
| 52 | +# Create VM instance |
| 53 | +limactl create --name=k8s ./contrib/vm.yaml |
| 54 | + |
| 55 | +# Start VM |
| 56 | +limactl start k8s |
| 57 | + |
| 58 | +# Use kubeconfig provided by VM |
| 59 | +export KUBECONFIG="${HOME}/.lima/k8s/copied-from-guest/kubeconfig.yaml" |
| 60 | +``` |
| 61 | + |
| 62 | +### Install Prerequisites |
| 63 | + |
| 64 | +```bash |
| 65 | +# Install cert-manager |
| 66 | +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.2/cert-manager.yaml |
| 67 | + |
| 68 | +# Install Rook operator |
| 69 | +kubectl apply -f ./rook/deploy/examples/crds.yaml |
| 70 | +kubectl apply -f ./rook/deploy/examples/common.yaml |
| 71 | +kubectl apply -f ./rook/deploy/examples/operator.yaml |
| 72 | +kubectl apply -f ./rook/deploy/examples/csi-operator.yaml |
| 73 | + |
| 74 | +# Create Ceph cluster |
| 75 | +kubectl apply -f ./rook/deploy/examples/cluster-test.yaml |
| 76 | + |
| 77 | +# (Optional) Install Ceph toolbox |
| 78 | +kubectl apply -f ./rook/deploy/examples/toolbox.yaml |
| 79 | +``` |
| 80 | + |
| 81 | +### Build and Install Operator |
| 82 | + |
| 83 | +```bash |
| 84 | +# Build image |
| 85 | +limactl shell k8s sudo nerdctl --namespace k8s.io build \ |
| 86 | + -t localhost:5000/cobaltcore-dev/external-arbiter-operator:latest \ |
| 87 | + -f ./Dockerfile . |
| 88 | + |
| 89 | +# Dry run operator install via Helm |
| 90 | +helm install --dry-run --create-namespace --namespace arbiter-operator \ |
| 91 | + --values ./contrib/charts/external-arbiter-operator/local.yaml \ |
| 92 | + arbiter-operator ./contrib/charts/external-arbiter-operator |
| 93 | + |
| 94 | +# Install operator via Helm chart |
| 95 | +helm install --create-namespace --namespace arbiter-operator \ |
| 96 | + --values ./contrib/charts/external-arbiter-operator/local.yaml \ |
| 97 | + arbiter-operator ./contrib/charts/external-arbiter-operator |
| 98 | +``` |
| 99 | + |
| 100 | +### Configure and Deploy Arbiter |
| 101 | + |
| 102 | +```bash |
| 103 | +# Create namespace, user, role, rolebinding, kubeconfig and secret for arbiter |
| 104 | +./hack/configure-k8s-user.sh |
| 105 | + |
| 106 | +# Create secret with remote cluster access configuration |
| 107 | +kubectl apply -f ./contrib/k8s/examples/secret.yaml -n arbiter-operator |
| 108 | + |
| 109 | +# Create remote cluster resource |
| 110 | +kubectl apply -f ./contrib/k8s/examples/remote-cluster.yaml -n arbiter-operator |
| 111 | + |
| 112 | +# Create remote arbiter resource |
| 113 | +kubectl apply -f ./contrib/k8s/examples/remote-arbiter.yaml -n arbiter-operator |
| 114 | + |
| 115 | +# Watch until Arbiter is ready |
| 116 | +kubectl get remotearbiter -n arbiter-operator -w |
| 117 | + |
| 118 | +# Check that Arbiter has joined quorum |
| 119 | +kubectl exec deployment/rook-ceph-tools -n rook-ceph -it -- ceph mon dump |
| 120 | +``` |
| 121 | + |
| 122 | +### Cleanup |
| 123 | + |
| 124 | +```bash |
| 125 | +# Remove Helm chart |
| 126 | +helm uninstall --namespace arbiter-operator arbiter-operator |
| 127 | + |
| 128 | +# Stop VM |
| 129 | +limactl stop k8s |
| 130 | + |
| 131 | +# Delete VM |
| 132 | +limactl delete k8s |
| 133 | +``` |
| 134 | + |
| 135 | +## Make Goals |
| 136 | + |
| 137 | +Useful make commands for development: |
| 138 | + |
| 139 | +```bash |
| 140 | +# Build binary |
| 141 | +make |
| 142 | + |
| 143 | +# Prettify project, run linters, etc. |
| 144 | +make pretty |
| 145 | + |
| 146 | +# Run tests |
| 147 | +make test |
| 148 | + |
| 149 | +# Regenerate Kubernetes resources |
| 150 | +make gen |
| 151 | + |
| 152 | +# Copy CRD definitions to Helm chart |
| 153 | +make helm |
| 154 | +``` |
| 155 | + |
| 156 | +## Configuration |
| 157 | + |
| 158 | +### Deployment Configuration |
| 159 | + |
| 160 | +Deployment manifests are managed by Helm. The `values.yaml` file lists all |
| 161 | +available configuration options. |
| 162 | + |
| 163 | +### Resource Configuration |
| 164 | + |
| 165 | +The following example resources are provided: |
| 166 | + |
| 167 | +- `secret.yaml` - Kubeconfig secret for arbiter installation |
| 168 | +- `remote-cluster.yaml` - RemoteCluster resource definition |
| 169 | +- `remote-arbiter.yaml` - RemoteArbiter resource definition |
| 170 | + |
| 171 | +## How to Run |
| 172 | + |
| 173 | +### Prerequisites |
| 174 | + |
| 175 | +Before running the operator, ensure the following conditions are met: |
| 176 | + |
| 177 | +1. A Ceph cluster operated by Rook is already up and running on the source |
| 178 | + Kubernetes cluster |
| 179 | +1. Resources (pods, services) from the target (arbiter) cluster are reachable |
| 180 | + from the source (operator/Rook) cluster and vice versa |
| 181 | + |
| 182 | +### Deployment Steps |
| 183 | + |
| 184 | +1. Create a user on the target cluster. |
| 185 | +1. Create the target namespace on the target cluster. |
| 186 | +1. Grant the user permissions to manage deployments, secrets, configmaps, their |
| 187 | + statuses, and finalizers. |
| 188 | +1. Provision the target user kubeconfig on the source cluster via secret. |
| 189 | +1. Deploy the operator on the source cluster. |
| 190 | +1. Create a RemoteCluster resource on the source cluster, referencing the target |
| 191 | + user kubeconfig secret. |
| 192 | +1. Create a RemoteArbiter resource on the source cluster, referencing the |
| 193 | + RemoteCluster. |
| 194 | +1. Watch until resources are ready. |
| 195 | +1. Verify that the arbiter has joined the quorum by running `ceph mon dump`. |
| 196 | + |
| 197 | +## See Also |
| 198 | +[Arbiter Repository](https://github.com/cobaltcore-dev/external-arbiter-operator?tab=readme-ov-file) |
0 commit comments