Skip to content

Commit 241883d

Browse files
authored
Merge pull request #10 from zdover23/docs-2026-03-12-docs-architecture-arbiter
Add "Arbiter" document to Architecture
2 parents f6d3249 + 2689197 commit 241883d

1 file changed

Lines changed: 198 additions & 0 deletions

File tree

docs/architecture/arbiter.md

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
---
2+
title: Arbiter
3+
---
4+
5+
# Arbiter
6+
7+
## About This Project
8+
9+
The external-arbiter-operator (Arbiter) works with Rook-provisioned Ceph
10+
clusters to deploy external arbiters (monitors) that are not managed by Rook
11+
but that participate in consensus.
12+
13+
The operator also monitors the remote cluster to verify its availability and
14+
ensure that the tenant has sufficient permissions to handle the deployment of
15+
Arbiter.
16+
17+
## Requirements and Setup
18+
19+
### Required Tools
20+
21+
The following tools are required on your development machine:
22+
23+
- `sed`
24+
- `openssl`
25+
- `make`
26+
- `git`
27+
- `golang`
28+
- `lima` (or another method to provision Kubernetes locally, such as Minikube)
29+
- `kubectl`
30+
- `docker` (or any compatible container engine, such as Podman)
31+
- `helm`
32+
33+
The remaining dependencies are provisioned via Go tools, including the
34+
Kubebuilder toolset.
35+
36+
## Quick Start
37+
38+
What follows is a quick walkthrough on how to prepare the environment, run the
39+
operator locally, and deploy an external monitor.
40+
41+
### Clone and Setup
42+
43+
```bash
44+
# Clone the Rook repository: https://github.com/rook/rook
45+
46+
#Run `make deps`:
47+
make deps
48+
49+
# Create OSD for Ceph
50+
limactl disk create osd --size=8G
51+
52+
# Create VM instance
53+
limactl create --name=k8s ./contrib/vm.yaml
54+
55+
# Start VM
56+
limactl start k8s
57+
58+
# Use kubeconfig provided by VM
59+
export KUBECONFIG="${HOME}/.lima/k8s/copied-from-guest/kubeconfig.yaml"
60+
```
61+
62+
### Install Prerequisites
63+
64+
```bash
65+
# Install cert-manager
66+
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.2/cert-manager.yaml
67+
68+
# Install Rook operator
69+
kubectl apply -f ./rook/deploy/examples/crds.yaml
70+
kubectl apply -f ./rook/deploy/examples/common.yaml
71+
kubectl apply -f ./rook/deploy/examples/operator.yaml
72+
kubectl apply -f ./rook/deploy/examples/csi-operator.yaml
73+
74+
# Create Ceph cluster
75+
kubectl apply -f ./rook/deploy/examples/cluster-test.yaml
76+
77+
# (Optional) Install Ceph toolbox
78+
kubectl apply -f ./rook/deploy/examples/toolbox.yaml
79+
```
80+
81+
### Build and Install Operator
82+
83+
```bash
84+
# Build image
85+
limactl shell k8s sudo nerdctl --namespace k8s.io build \
86+
-t localhost:5000/cobaltcore-dev/external-arbiter-operator:latest \
87+
-f ./Dockerfile .
88+
89+
# Dry run operator install via Helm
90+
helm install --dry-run --create-namespace --namespace arbiter-operator \
91+
--values ./contrib/charts/external-arbiter-operator/local.yaml \
92+
arbiter-operator ./contrib/charts/external-arbiter-operator
93+
94+
# Install operator via Helm chart
95+
helm install --create-namespace --namespace arbiter-operator \
96+
--values ./contrib/charts/external-arbiter-operator/local.yaml \
97+
arbiter-operator ./contrib/charts/external-arbiter-operator
98+
```
99+
100+
### Configure and Deploy Arbiter
101+
102+
```bash
103+
# Create namespace, user, role, rolebinding, kubeconfig and secret for arbiter
104+
./hack/configure-k8s-user.sh
105+
106+
# Create secret with remote cluster access configuration
107+
kubectl apply -f ./contrib/k8s/examples/secret.yaml -n arbiter-operator
108+
109+
# Create remote cluster resource
110+
kubectl apply -f ./contrib/k8s/examples/remote-cluster.yaml -n arbiter-operator
111+
112+
# Create remote arbiter resource
113+
kubectl apply -f ./contrib/k8s/examples/remote-arbiter.yaml -n arbiter-operator
114+
115+
# Watch until Arbiter is ready
116+
kubectl get remotearbiter -n arbiter-operator -w
117+
118+
# Check that Arbiter has joined quorum
119+
kubectl exec deployment/rook-ceph-tools -n rook-ceph -it -- ceph mon dump
120+
```
121+
122+
### Cleanup
123+
124+
```bash
125+
# Remove Helm chart
126+
helm uninstall --namespace arbiter-operator arbiter-operator
127+
128+
# Stop VM
129+
limactl stop k8s
130+
131+
# Delete VM
132+
limactl delete k8s
133+
```
134+
135+
## Make Goals
136+
137+
Useful make commands for development:
138+
139+
```bash
140+
# Build binary
141+
make
142+
143+
# Prettify project, run linters, etc.
144+
make pretty
145+
146+
# Run tests
147+
make test
148+
149+
# Regenerate Kubernetes resources
150+
make gen
151+
152+
# Copy CRD definitions to Helm chart
153+
make helm
154+
```
155+
156+
## Configuration
157+
158+
### Deployment Configuration
159+
160+
Deployment manifests are managed by Helm. The `values.yaml` file lists all
161+
available configuration options.
162+
163+
### Resource Configuration
164+
165+
The following example resources are provided:
166+
167+
- `secret.yaml` - Kubeconfig secret for arbiter installation
168+
- `remote-cluster.yaml` - RemoteCluster resource definition
169+
- `remote-arbiter.yaml` - RemoteArbiter resource definition
170+
171+
## How to Run
172+
173+
### Prerequisites
174+
175+
Before running the operator, ensure the following conditions are met:
176+
177+
1. A Ceph cluster operated by Rook is already up and running on the source
178+
Kubernetes cluster
179+
1. Resources (pods, services) from the target (arbiter) cluster are reachable
180+
from the source (operator/Rook) cluster and vice versa
181+
182+
### Deployment Steps
183+
184+
1. Create a user on the target cluster.
185+
1. Create the target namespace on the target cluster.
186+
1. Grant the user permissions to manage deployments, secrets, configmaps, their
187+
statuses, and finalizers.
188+
1. Provision the target user kubeconfig on the source cluster via secret.
189+
1. Deploy the operator on the source cluster.
190+
1. Create a RemoteCluster resource on the source cluster, referencing the target
191+
user kubeconfig secret.
192+
1. Create a RemoteArbiter resource on the source cluster, referencing the
193+
RemoteCluster.
194+
1. Watch until resources are ready.
195+
1. Verify that the arbiter has joined the quorum by running `ceph mon dump`.
196+
197+
## See Also
198+
[Arbiter Repository](https://github.com/cobaltcore-dev/external-arbiter-operator?tab=readme-ov-file)

0 commit comments

Comments
 (0)