Skip to content

deploy.md

Latest commit

 

History

History
211 lines (153 loc) · 8.07 KB

deploy.md

File metadata and controls

211 lines (153 loc) · 8.07 KB
Raw

Deploying Nebula

This guide covers building and deploying the Nebula manager to a Kubernetes cluster, including wiring provider credentials.

Quick start

# 1. Put provider credentials in .env (secrets only — gitignored).
cp .env.example .env
$EDITOR .env            # set MODAL_TOKEN_ID / MODAL_TOKEN_SECRET (from `modal token new`)

# 2. Build, apply credential Secrets, and deploy in one step.
make deploy-all IMG=<your-registry>/nebula:<tag>

For a local Kind cluster, load the image instead of pushing it to a registry:

make deploy-all IMG=nebula:dev DEPLOY_KIND_CLUSTER=<kind-cluster-name>

That's it. The manager comes up in the nebula-system namespace and registers every provider whose credentials are present.

How credentials are handled

Nebula keeps provider credentials in one Kubernetes Secret per provider, each referenced by the manager Deployment through an optional envFrom.secretRef (see config/manager/manager.yaml). So:

  • The manager boots even when a Secret is missing.
  • A provider whose credentials are absent is logged and skipped at registration, not fatal. A NodePool referencing it surfaces Ready=False / UnknownProvider, which self-heals once the creds are applied.

.env holds only secrets. Non-secret configuration (image, namespace, Kind cluster) is passed as make variables.

A typo in a Secret name looks exactly like an absent one — the provider is silently skipped. UnknownProvider on the NodePool is your signal.

Webhook TLS (no cert-manager)

Nebula runs a mutating webhook (it injects the scheduling gate into gated Pods), and the API server requires TLS to call it. cert-manager is intentionally not used, and neither is any out-of-band setup step: the manager provisions its own serving certificate in-process at startup (pkg/cert, built on cert-controller). There is nothing to install and nothing to run before make deploy.

At startup it mints a serving cert into the nebula-webhook-server-cert Secret, patches the matching CA into the MutatingWebhookConfiguration caBundle, and rotates the cert before it expires.

Two things this implies for the deployment:

  • The volume at /tmp/k8s-webhook-server/serving-certs must be that Secret, not an emptyDir — the rotator writes only the Secret, and the kubelet projects it. With an emptyDir the files never appear and nothing starts, while the pod still looks healthy.
  • The first seconds of a fresh install log waiting for the webhook certificate to be ready and reconcile nothing. That is expected: controllers wait for the cert so no Pod is admitted before the webhook is trusted.

To use cert-manager instead, re-add - ../certmanager and re-enable the CERTMANAGER replacements blocks in config/default/kustomization.yaml, install cert-manager, and drop the CertsManager call from cmd/main.go.

What deploy-all does

make deploy-all runs hack/deploy.sh, which is idempotent (safe to re-run):

  1. Builds the manager image (make docker-build IMG=…).
  2. Publishes it — docker push, or kind load docker-image when DEPLOY_KIND_CLUSTER is set.
  3. Creates the namespace and credential Secrets from .env, one per provider, before deploying — so the manager boots already-configured. A Secret is created only when all its required keys are set; a provider with blank keys is skipped.
  4. Deploys CRDs + the manager (make deploy IMG=…).

There is no webhook-cert step and no CA-bundle injection step — the manager does both itself at startup (see Webhook TLS).

Configuration

.env (secrets only, gitignored — see .env.example):

Key Provider Required Notes
MODAL_TOKEN_ID Modal yes From modal token new
MODAL_TOKEN_SECRET Modal yes From modal token new
AWS_ACCESS_KEY_ID AWS dev only Prefer IRSA / instance role in production and leave blank — the SDK's default credential chain finds the role. Set only for local/dev.
AWS_SECRET_ACCESS_KEY AWS dev only Pairs with AWS_ACCESS_KEY_ID; both required together or both blank.

Non-secret config, passed as make variables:

Variable Default Meaning
IMG controller:latest Manager image to build and deploy
NAMESPACE nebula-system Namespace the manager runs in
DEPLOY_KIND_CLUSTER (empty) If set, load the image into this Kind cluster instead of pushing. Separate from the e2e KIND_CLUSTER, so a plain make deploy-all pushes.
KUBECTL kubectl kubectl binary to use

Manual deployment

If you don't want the script (e.g. you manage Secrets via sealed-secrets or a GitOps pipeline), do the same steps by hand. Order matters — create the Secrets before deploying so the manager boots configured:

# 1. Namespace.
kubectl create namespace nebula-system --dry-run=client -o yaml | kubectl apply -f -

# 2. Modal credential Secret (before deploy — read as env at pod startup).
kubectl create secret generic nebula-modal-credentials -n nebula-system \
  --from-literal=MODAL_TOKEN_ID=ak-... \
  --from-literal=MODAL_TOKEN_SECRET=as-...

# 3. Deploy CRDs + manager. The pod reads creds on first boot, and provisions its
#    own webhook cert + caBundle at startup — no cert step of your own.
make deploy IMG=<your-registry>/nebula:<tag>

No restart is needed — everything the manager consumes exists before it boots. If you instead created a Secret after the manager was already running, restart it to pick up the change: kubectl rollout restart deployment/nebula-controller-manager -n nebula-system.

A declarative Secret manifest for GitOps workflows lives at config/manager/modal-credentials.example.yaml (not applied by kustomize — copy and fill it in, or template it with your secret manager).

To tear everything down:

make undeploy      # remove the manager
make uninstall     # remove CRDs

Verifying the deployment

# Manager is running.
kubectl -n nebula-system get pods

# Providers registered (look for "registered provider" / "skipping … registration").
kubectl -n nebula-system logs deploy/nebula-controller-manager | grep -i provider

# Virtual nodes exist, one per registered provider.
kubectl get nodes -l nebula.inftyai.com/provider

# Webhook TLS is wired: the caBundle matches the serving cert Secret.
diff <(kubectl get secret nebula-webhook-server-cert -n nebula-system -o jsonpath='{.data.tls\.crt}') \
     <(kubectl get mutatingwebhookconfiguration nebula-mutating-webhook-configuration \
         -o jsonpath='{.webhooks[0].clientConfig.caBundle}') \
  && echo "webhook caBundle matches serving cert"

A pool referencing an unregistered provider shows it plainly:

kubectl get nodepool <name> -o jsonpath='{.status.conditions}'
# Ready=False / UnknownProvider means that provider's creds are missing or wrong.

Smoke test

The checks above only prove the manager is healthy. To confirm a provider can actually launch — this starts a real GPU instance and costs money:

kubectl apply -f config/samples/nodepool.yaml
kubectl apply -f config/samples/sandbox.yaml

kubectl get sandbox sample -w      # Pending → Initializing → Ready
kubectl describe sandbox sample    # events name the provider, region and instance type

kubectl delete -f config/samples/sandbox.yaml   # terminates the instance

Both samples are commented with what the fields mean and which accelerator shapes actually resolve. If the box never reaches Ready, check your provider GPU quota (on a new AWS account it is often 0, and On-Demand and Spot are separate limits).