From 08ed2d7cb5d03be7b73349879aadc186dbed5c2f Mon Sep 17 00:00:00 2001 From: clyi Date: Tue, 21 Jul 2026 16:49:23 +0800 Subject: [PATCH 1/6] docs: add VPC egress gateway BFD mitigation guide --- ...itigating_VPC_Egress_Gateway_BFD_issues.md | 186 +++++++++++++++++ ...itigating_VPC_Egress_Gateway_BFD_issues.md | 187 ++++++++++++++++++ 2 files changed, 373 insertions(+) create mode 100644 docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md create mode 100644 docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md diff --git a/docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md b/docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md new file mode 100644 index 000000000..12eb2a8b6 --- /dev/null +++ b/docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md @@ -0,0 +1,186 @@ +--- +products: + - Alauda Container Platform +kind: + - Solution +ProductsVersion: + - 4.3.x +id: TBD +--- + +# Temporary solution for VPC Egress Gateway BFD issues + +## Issue + +A VPC Egress Gateway (VEG) has BFD enabled, but application Pods selected by the VEG cannot access external networks. The VEG Pod can remain `Running` and `Ready` while the local BFD session is unavailable. + +When the issue occurs, `bfdd-control status` can report an empty session table: + +```text +There are 0 sessions: +``` + +The BFD session might also frequently transition between `Up` and `Down`. Without a stable session, the corresponding VEG next hop can become unavailable and interrupt application outbound traffic. + +## Environment + +- Alauda Container Platform 4.3.x +- VPC Egress Gateway with BFD enabled +- `kubectl` access to the workload cluster + +For information about configuring a VPC Egress Gateway and enabling BFD, see [Alauda Container Platform Egress Gateway Installation and Usage Guide](./Alauda_Container_Platform_Egress_Gateway_Installation_and_Usage_Guide.md). + +## Cause + +The existing `bfdd` health probe runs `bfdd-control status` directly. The command returns exit code `0` when it successfully communicates with the local daemon, even if the daemon reports no BFD sessions: + +| BFD state | Example output | Exit code | +| --- | --- | --- | +| A session exists | `There are 1 sessions:` | `0` | +| No session exists | `There are 0 sessions:` | `0` | + +Kubernetes exec probes evaluate the exit code and do not interpret the command output. Therefore, the Pod remains healthy when the local session table is empty, and the kubelet does not restart `bfdd` to reinitialize its BFD configuration. + +The following settings can also increase the likelihood of BFD instability in some environments: + +- In ACP 4.3.0 and 4.3.1, the default `bfdd` CPU request and limit are `50m`. CPU throttling can delay BFD packet processing. +- Some VEG configurations based on the installation guide use `100ms` for both `minRX` and `minTX`. This interval can be sensitive to short scheduling delays or network jitter. + +These settings are possible contributing factors rather than a confirmed root cause for every BFD failure. + +## Resolution + +Modify the `bfdd` liveness probe so that it detects `There are 0 sessions:` and lets the kubelet restart the container after three consecutive failures. If the environment also requires the optional CPU or BFD interval adjustments, apply those VEG changes first and apply the Deployment probe last. + +Set the VEG namespace and name, and obtain the generated Deployment name: + +```bash +NAMESPACE="" +VEG_NAME="" + +DEPLOYMENT="$(kubectl -n "${NAMESPACE}" get deployment \ + -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ + -o jsonpath='{.items[0].metadata.name}')" +``` + +:::warning +Changing the VEG settings or Deployment recreates VEG Pods. For a single-replica VEG, schedule a maintenance window because outbound traffic can be interrupted during the rollout. +::: + +### Optional: Increase `bfdd` CPU + +For ACP 4.3.0 and 4.3.1, or when `bfdd` uses a `50m` CPU request or limit and the BFD session frequently flaps, increase both values to `200m`: + +Edit the VEG resource: + +```bash +kubectl -n "${NAMESPACE}" edit veg "${VEG_NAME}" +``` + +Update only the CPU fields under `spec.resources`. Preserve all other existing resource settings: + +```yaml +spec: + resources: + requests: + cpu: 200m + limits: + cpu: 200m +``` + +Wait for the generated Deployment to finish rolling out: + +```bash +kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ + --timeout=180s +``` + +### Optional: Increase the BFD intervals + +If both `minRX` and `minTX` are `100ms` and the BFD session flaps during short scheduling delays or network jitter, increase them to `1000ms`: + +Edit the VEG resource: + +```bash +kubectl -n "${NAMESPACE}" edit veg "${VEG_NAME}" +``` + +Update only `minRX` and `minTX` under `spec.bfd`. Preserve the other BFD settings: + +```yaml +spec: + bfd: + minRX: 1000 + minTX: 1000 +``` + +Wait for the generated Deployment to finish rolling out: + +```bash +kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ + --timeout=180s +``` + +With `multiplier: 5`, this change increases the approximate failure detection time from `500ms` to `5s`. If the VEG already uses `1000ms`, no change is required. + +### Apply the temporary liveness probe + +Edit the generated Deployment: + +```bash +kubectl -n "${NAMESPACE}" edit deployment "${DEPLOYMENT}" +``` + +Find the container named `bfdd` and replace only its existing `livenessProbe` with the following configuration. Preserve the other container and Deployment fields: + +```yaml +livenessProbe: + exec: + command: + - bash + - -ec + - | + output="$(bfdd-control status)" + + if grep -q '^There are 0 sessions:' <<< "${output}"; then + exit 1 + fi + initialDelaySeconds: 15 + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 3 + successThreshold: 1 +``` + +Wait for the rollout to finish: + +```bash +kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ + --timeout=180s +``` + +Keep the existing readiness probe unchanged. Making readiness depend on the BFD session being `Up` can prevent a new or recovering VEG Pod from being selected while the BFD session is being established. + +## Verification + +Check the BFD session and `bfdd` restart count in each VEG Pod: + +```bash +for pod in $(kubectl -n "${NAMESPACE}" get pods \ + -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ + -o jsonpath='{.items[*].metadata.name}'); do + echo "=== ${pod} ===" + kubectl -n "${NAMESPACE}" exec "${pod}" -c bfdd -- bfdd-control status + kubectl -n "${NAMESPACE}" get pod "${pod}" \ + -o jsonpath='bfdd restartCount={.status.containerStatuses[?(@.name=="bfdd")].restartCount}{"\n"}' +done +``` + +After the liveness probe detects `There are 0 sessions:` three consecutive times, the kubelet restarts `bfdd`. The existing startup probe then configures the BFD parameters and registers the peers again. Verify that a session is created and eventually reaches `Up`. + +## Notes + +- The temporary liveness probe handles only an empty local session table. It does not repair an existing session that remains `Down`. +- If `bfdd` keeps restarting or the session remains `Down`, investigate the VPC BFD port, OVN-side BFD state, peer reachability, and the network path. +- The VEG Deployment is generated by the Kube-OVN controller. The manual probe change can be overwritten when the VEG specification or replica count changes, Kube-OVN is upgraded, or the VEG or Deployment is recreated. +- Increasing the BFD interval to `1000ms` reduces sensitivity to short interruptions but delays detection and failover for real failures. diff --git a/docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md b/docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md new file mode 100644 index 000000000..7fef1daa6 --- /dev/null +++ b/docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md @@ -0,0 +1,187 @@ +--- +products: + - Alauda Container Platform +kind: + - Solution +ProductsVersion: + - 4.3.x +id: TBD +sourceSHA: 08211c0ca1745bac1c5bf7895bbd30cd98a4f3c66c59bff6d6cd726edc398cfa +--- + +# VPC 出口网关 BFD 异常的临时解决方案 + +## 问题现象 + +VPC 出口网关（VEG）已启用 BFD，但被 VEG 选中的业务 Pod 无法访问外部网络。此时 VEG Pod 可能仍处于 `Running` 和 `Ready` 状态，但本地 BFD 会话已经不可用。 + +问题发生时，`bfdd-control status` 可能显示本地会话表为空： + +```text +There are 0 sessions: +``` + +BFD 会话也可能频繁在 `Up` 和 `Down` 之间切换。BFD 会话不稳定时，对应的 VEG 下一跳可能变为不可用，导致业务出网流量中断。 + +## 环境 + +- Alauda Container Platform 4.3.x +- 已启用 BFD 的 VPC 出口网关 +- 可以使用 `kubectl` 访问业务集群 + +有关配置 VPC 出口网关和启用 BFD 的方法，请参阅 [Alauda Container Platform 出口网关安装与使用指南](./Alauda_Container_Platform_Egress_Gateway_Installation_and_Usage_Guide.md)。 + +## 原因 + +现有 `bfdd` 健康探针直接执行 `bfdd-control status`。该命令只要能够成功连接本地守护进程，就会返回退出码 `0`，即使本地已经不存在 BFD 会话： + +| BFD 状态 | 示例输出 | 退出码 | +| --- | --- | --- | +| 存在 BFD 会话 | `There are 1 sessions:` | `0` | +| 不存在 BFD 会话 | `There are 0 sessions:` | `0` | + +Kubernetes exec 探针只判断命令退出码，不解析命令输出。因此，本地会话表为空时 Pod 仍然保持健康，kubelet 不会重启 `bfdd` 并重新初始化 BFD 配置。 + +以下配置也可能增加部分环境中 BFD 不稳定的概率： + +- ACP 4.3.0 和 4.3.1 中，`bfdd` 默认的 CPU request 和 limit 为 `50m`。CPU 限流可能延迟 BFD 报文处理。 +- 部分参照安装指南创建的 VEG 将 `minRX` 和 `minTX` 设置为 `100ms`，该间隔可能对短时调度延迟或网络抖动过于敏感。 + +这些配置是可能的影响因素，不代表所有 BFD 异常都由它们导致。 + +## 解决方案 + +修改 `bfdd` 存活探针，使其能够识别 `There are 0 sessions:`。连续三次检查失败后，由 kubelet 重启容器。如果环境还需要调整 CPU 或 BFD 间隔，请先修改 VEG 配置，最后再修改 Deployment 探针。 + +设置 VEG 所在的命名空间和名称，并获取控制器生成的 Deployment 名称： + +```bash +NAMESPACE="" +VEG_NAME="" + +DEPLOYMENT="$(kubectl -n "${NAMESPACE}" get deployment \ + -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ + -o jsonpath='{.items[0].metadata.name}')" +``` + +:::warning +修改 VEG 配置或 Deployment 会重建 VEG Pod。对于单副本 VEG，请安排维护窗口，因为滚动更新期间业务出网可能暂时中断。 +::: + +### 调整 `bfdd` CPU（可选） + +对于 ACP 4.3.0 和 4.3.1，或者 `bfdd` CPU request 或 limit 为 `50m` 且 BFD 会话频繁翻转的环境，可以将两个值调整为 `200m`： + +编辑 VEG 资源： + +```bash +kubectl -n "${NAMESPACE}" edit veg "${VEG_NAME}" +``` + +只修改 `spec.resources` 下的 CPU 字段，保留其他已有资源配置： + +```yaml +spec: + resources: + requests: + cpu: 200m + limits: + cpu: 200m +``` + +等待控制器生成的 Deployment 完成滚动更新： + +```bash +kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ + --timeout=180s +``` + +### 调整 BFD 检测间隔（可选） + +如果 `minRX` 和 `minTX` 均为 `100ms`，并且 BFD 会话在短时调度延迟或网络抖动期间频繁翻转，可以将两个值调整为 `1000ms`： + +编辑 VEG 资源： + +```bash +kubectl -n "${NAMESPACE}" edit veg "${VEG_NAME}" +``` + +只修改 `spec.bfd` 下的 `minRX` 和 `minTX`，保留其他 BFD 配置： + +```yaml +spec: + bfd: + minRX: 1000 + minTX: 1000 +``` + +等待控制器生成的 Deployment 完成滚动更新： + +```bash +kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ + --timeout=180s +``` + +当 `multiplier: 5` 时，该调整会将近似故障检测时间从 `500ms` 延长到 `5s`。如果 VEG 已经使用 `1000ms`，则无需修改。 + +### 应用临时存活探针 + +编辑控制器生成的 Deployment： + +```bash +kubectl -n "${NAMESPACE}" edit deployment "${DEPLOYMENT}" +``` + +找到名为 `bfdd` 的容器，只将其现有 `livenessProbe` 替换为以下配置，保留容器和 Deployment 中的其他字段： + +```yaml +livenessProbe: + exec: + command: + - bash + - -ec + - | + output="$(bfdd-control status)" + + if grep -q '^There are 0 sessions:' <<< "${output}"; then + exit 1 + fi + initialDelaySeconds: 15 + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 3 + successThreshold: 1 +``` + +等待滚动更新完成： + +```bash +kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ + --timeout=180s +``` + +保持现有就绪探针不变。让就绪状态依赖 BFD 会话进入 `Up`，可能导致新建或正在恢复的 VEG Pod 在建立 BFD 会话期间无法被控制器选中。 + +## 验证 + +检查每个 VEG Pod 的 BFD 会话和 `bfdd` 重启次数： + +```bash +for pod in $(kubectl -n "${NAMESPACE}" get pods \ + -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ + -o jsonpath='{.items[*].metadata.name}'); do + echo "=== ${pod} ===" + kubectl -n "${NAMESPACE}" exec "${pod}" -c bfdd -- bfdd-control status + kubectl -n "${NAMESPACE}" get pod "${pod}" \ + -o jsonpath='bfdd restartCount={.status.containerStatuses[?(@.name=="bfdd")].restartCount}{"\n"}' +done +``` + +存活探针连续三次检测到 `There are 0 sessions:` 后，kubelet 会重启 `bfdd`。现有启动探针随后会重新配置 BFD 参数并注册对端。确认新的 BFD 会话已经创建并最终进入 `Up` 状态。 + +## 注意事项 + +- 临时存活探针只能处理本地会话表为空的情况，不能修复已经存在但持续处于 `Down` 状态的会话。 +- 如果 `bfdd` 持续重启或会话持续处于 `Down`，需要检查 VPC BFD 端口、OVN 侧 BFD 状态、对端连通性和网络路径。 +- VEG Deployment 由 Kube-OVN 控制器生成。修改 VEG spec 或副本数、升级 Kube-OVN，或者重建 VEG 或 Deployment 时，手工修改的探针可能被覆盖。 +- 将 BFD 检测间隔调整为 `1000ms` 可以降低对短时抖动的敏感度，但也会延长真实故障的发现和切换时间。 From 19c913f3871d82722459121093b68dfcf522a550 Mon Sep 17 00:00:00 2001 From: clyi Date: Tue, 21 Jul 2026 17:02:15 +0800 Subject: [PATCH 2/6] docs: clarify affected ACP versions --- .../Mitigating_VPC_Egress_Gateway_BFD_issues.md | 9 ++++++--- .../Mitigating_VPC_Egress_Gateway_BFD_issues.md | 11 +++++++---- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md b/docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md index 12eb2a8b6..8dc839ad8 100644 --- a/docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md +++ b/docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md @@ -4,7 +4,7 @@ products: kind: - Solution ProductsVersion: - - 4.3.x + - 4.3.0,4.3.1,4.3.2 id: TBD --- @@ -24,7 +24,7 @@ The BFD session might also frequently transition between `Up` and `Down`. Withou ## Environment -- Alauda Container Platform 4.3.x +- Alauda Container Platform 4.3.0, 4.3.1, or 4.3.2 - VPC Egress Gateway with BFD enabled - `kubectl` access to the workload cluster @@ -50,7 +50,9 @@ These settings are possible contributing factors rather than a confirmed root ca ## Resolution -Modify the `bfdd` liveness probe so that it detects `There are 0 sessions:` and lets the kubelet restart the container after three consecutive failures. If the environment also requires the optional CPU or BFD interval adjustments, apply those VEG changes first and apply the Deployment probe last. +Upgrade the cluster to ACP 4.3.3 or later, which includes the fix for this issue. After the upgrade, the temporary changes described below are not required. + +If the cluster must remain on ACP 4.3.0, 4.3.1, or 4.3.2, use the following temporary solution. Modify the `bfdd` liveness probe so that it detects `There are 0 sessions:` and lets the kubelet restart the container after three consecutive failures. If the environment also requires the optional CPU or BFD interval adjustments, apply those VEG changes first and apply the Deployment probe last. Set the VEG namespace and name, and obtain the generated Deployment name: @@ -180,6 +182,7 @@ After the liveness probe detects `There are 0 sessions:` three consecutive times ## Notes +- Do not apply these temporary changes to ACP 4.3.3 or later. Upgrade to a fixed release instead. - The temporary liveness probe handles only an empty local session table. It does not repair an existing session that remains `Down`. - If `bfdd` keeps restarting or the session remains `Down`, investigate the VPC BFD port, OVN-side BFD state, peer reachability, and the network path. - The VEG Deployment is generated by the Kube-OVN controller. The manual probe change can be overwritten when the VEG specification or replica count changes, Kube-OVN is upgraded, or the VEG or Deployment is recreated. diff --git a/docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md b/docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md index 7fef1daa6..dc885c60a 100644 --- a/docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md +++ b/docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md @@ -4,9 +4,9 @@ products: kind: - Solution ProductsVersion: - - 4.3.x + - 4.3.0,4.3.1,4.3.2 id: TBD -sourceSHA: 08211c0ca1745bac1c5bf7895bbd30cd98a4f3c66c59bff6d6cd726edc398cfa +sourceSHA: 97c9e9981ecf52d5197ef7124ba6bb85521469cf5b9d9ba9727dbeaa1c4c863a --- # VPC 出口网关 BFD 异常的临时解决方案 @@ -25,7 +25,7 @@ BFD 会话也可能频繁在 `Up` 和 `Down` 之间切换。BFD 会话不稳定 ## 环境 -- Alauda Container Platform 4.3.x +- Alauda Container Platform 4.3.0、4.3.1 或 4.3.2 - 已启用 BFD 的 VPC 出口网关 - 可以使用 `kubectl` 访问业务集群 @@ -51,7 +51,9 @@ Kubernetes exec 探针只判断命令退出码，不解析命令输出。因此 ## 解决方案 -修改 `bfdd` 存活探针，使其能够识别 `There are 0 sessions:`。连续三次检查失败后，由 kubelet 重启容器。如果环境还需要调整 CPU 或 BFD 间隔，请先修改 VEG 配置，最后再修改 Deployment 探针。 +将集群升级到 ACP 4.3.3 或更高版本，该版本已包含此问题的修复。升级后不需要应用下述临时修改。 + +如果集群暂时必须继续使用 ACP 4.3.0、4.3.1 或 4.3.2，请使用以下临时方案。修改 `bfdd` 存活探针，使其能够识别 `There are 0 sessions:`。连续三次检查失败后，由 kubelet 重启容器。如果环境还需要调整 CPU 或 BFD 间隔，请先修改 VEG 配置，最后再修改 Deployment 探针。 设置 VEG 所在的命名空间和名称，并获取控制器生成的 Deployment 名称： @@ -181,6 +183,7 @@ done ## 注意事项 +- ACP 4.3.3 或更高版本不需要应用这些临时修改，应直接升级到已修复版本。 - 临时存活探针只能处理本地会话表为空的情况，不能修复已经存在但持续处于 `Down` 状态的会话。 - 如果 `bfdd` 持续重启或会话持续处于 `Down`，需要检查 VPC BFD 端口、OVN 侧 BFD 状态、对端连通性和网络路径。 - VEG Deployment 由 Kube-OVN 控制器生成。修改 VEG spec 或副本数、升级 Kube-OVN，或者重建 VEG 或 Deployment 时，手工修改的探针可能被覆盖。 From ce0ca91cba9938049ac42c322de31450f9c72e21 Mon Sep 17 00:00:00 2001 From: clyi Date: Tue, 21 Jul 2026 17:40:43 +0800 Subject: [PATCH 3/6] docs: preserve VEG resource defaults --- .../Mitigating_VPC_Egress_Gateway_BFD_issues.md | 7 ++++++- .../Mitigating_VPC_Egress_Gateway_BFD_issues.md | 9 +++++++-- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md b/docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md index 8dc839ad8..85e13fcd3 100644 --- a/docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md +++ b/docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md @@ -79,17 +79,22 @@ Edit the VEG resource: kubectl -n "${NAMESPACE}" edit veg "${VEG_NAME}" ``` -Update only the CPU fields under `spec.resources`. Preserve all other existing resource settings: +Check the existing `spec.resources` configuration. If it is not configured, add the complete resource block below. Kube-OVN uses the entire user-provided block and does not add the default memory or ephemeral-storage values after `spec.resources` is set: ```yaml spec: resources: requests: cpu: 200m + memory: 50Mi limits: cpu: 200m + memory: 50Mi + ephemeral-storage: 1Gi ``` +If the VEG already defines custom memory or ephemeral-storage values, preserve those values and change only the CPU request and limit to `200m`. + Wait for the generated Deployment to finish rolling out: ```bash diff --git a/docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md b/docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md index dc885c60a..98507142a 100644 --- a/docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md +++ b/docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md @@ -6,7 +6,7 @@ kind: ProductsVersion: - 4.3.0,4.3.1,4.3.2 id: TBD -sourceSHA: 97c9e9981ecf52d5197ef7124ba6bb85521469cf5b9d9ba9727dbeaa1c4c863a +sourceSHA: 382a755423f1324ac634ff4cb04f2f9b2899fc0211bf96a53776d0f36baaeeb1 --- # VPC 出口网关 BFD 异常的临时解决方案 @@ -80,17 +80,22 @@ DEPLOYMENT="$(kubectl -n "${NAMESPACE}" get deployment \ kubectl -n "${NAMESPACE}" edit veg "${VEG_NAME}" ``` -只修改 `spec.resources` 下的 CPU 字段，保留其他已有资源配置： +先检查现有 `spec.resources` 配置。如果尚未配置，请添加以下完整资源配置。设置 `spec.resources` 后，Kube-OVN 会直接使用用户提供的整个配置，不会继续补充默认的内存和临时存储值： ```yaml spec: resources: requests: cpu: 200m + memory: 50Mi limits: cpu: 200m + memory: 50Mi + ephemeral-storage: 1Gi ``` +如果 VEG 已经配置了自定义内存或临时存储值，请保留这些值，只将 CPU request 和 limit 调整为 `200m`。 + 等待控制器生成的 Deployment 完成滚动更新： ```bash From 04b04b700085f711af79701a84233386f8623650 Mon Sep 17 00:00:00 2001 From: clyi Date: Wed, 22 Jul 2026 14:56:39 +0800 Subject: [PATCH 4/6] docs: focus VEG BFD guide on zero sessions --- ...itigating_VPC_Egress_Gateway_BFD_issues.md | 194 ----------------- ...ay_remains_Ready_with_zero_BFD_sessions.md | 108 ++++++++++ ...itigating_VPC_Egress_Gateway_BFD_issues.md | 195 ------------------ ...ay_remains_Ready_with_zero_BFD_sessions.md | 109 ++++++++++ 4 files changed, 217 insertions(+), 389 deletions(-) delete mode 100644 docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md create mode 100644 docs/en/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md delete mode 100644 docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md create mode 100644 docs/zh/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md diff --git a/docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md b/docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md deleted file mode 100644 index 85e13fcd3..000000000 --- a/docs/en/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md +++ /dev/null @@ -1,194 +0,0 @@ ---- -products: - - Alauda Container Platform -kind: - - Solution -ProductsVersion: - - 4.3.0,4.3.1,4.3.2 -id: TBD ---- - -# Temporary solution for VPC Egress Gateway BFD issues - -## Issue - -A VPC Egress Gateway (VEG) has BFD enabled, but application Pods selected by the VEG cannot access external networks. The VEG Pod can remain `Running` and `Ready` while the local BFD session is unavailable. - -When the issue occurs, `bfdd-control status` can report an empty session table: - -```text -There are 0 sessions: -``` - -The BFD session might also frequently transition between `Up` and `Down`. Without a stable session, the corresponding VEG next hop can become unavailable and interrupt application outbound traffic. - -## Environment - -- Alauda Container Platform 4.3.0, 4.3.1, or 4.3.2 -- VPC Egress Gateway with BFD enabled -- `kubectl` access to the workload cluster - -For information about configuring a VPC Egress Gateway and enabling BFD, see [Alauda Container Platform Egress Gateway Installation and Usage Guide](./Alauda_Container_Platform_Egress_Gateway_Installation_and_Usage_Guide.md). - -## Cause - -The existing `bfdd` health probe runs `bfdd-control status` directly. The command returns exit code `0` when it successfully communicates with the local daemon, even if the daemon reports no BFD sessions: - -| BFD state | Example output | Exit code | -| --- | --- | --- | -| A session exists | `There are 1 sessions:` | `0` | -| No session exists | `There are 0 sessions:` | `0` | - -Kubernetes exec probes evaluate the exit code and do not interpret the command output. Therefore, the Pod remains healthy when the local session table is empty, and the kubelet does not restart `bfdd` to reinitialize its BFD configuration. - -The following settings can also increase the likelihood of BFD instability in some environments: - -- In ACP 4.3.0 and 4.3.1, the default `bfdd` CPU request and limit are `50m`. CPU throttling can delay BFD packet processing. -- Some VEG configurations based on the installation guide use `100ms` for both `minRX` and `minTX`. This interval can be sensitive to short scheduling delays or network jitter. - -These settings are possible contributing factors rather than a confirmed root cause for every BFD failure. - -## Resolution - -Upgrade the cluster to ACP 4.3.3 or later, which includes the fix for this issue. After the upgrade, the temporary changes described below are not required. - -If the cluster must remain on ACP 4.3.0, 4.3.1, or 4.3.2, use the following temporary solution. Modify the `bfdd` liveness probe so that it detects `There are 0 sessions:` and lets the kubelet restart the container after three consecutive failures. If the environment also requires the optional CPU or BFD interval adjustments, apply those VEG changes first and apply the Deployment probe last. - -Set the VEG namespace and name, and obtain the generated Deployment name: - -```bash -NAMESPACE="" -VEG_NAME="" - -DEPLOYMENT="$(kubectl -n "${NAMESPACE}" get deployment \ - -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ - -o jsonpath='{.items[0].metadata.name}')" -``` - -:::warning -Changing the VEG settings or Deployment recreates VEG Pods. For a single-replica VEG, schedule a maintenance window because outbound traffic can be interrupted during the rollout. -::: - -### Optional: Increase `bfdd` CPU - -For ACP 4.3.0 and 4.3.1, or when `bfdd` uses a `50m` CPU request or limit and the BFD session frequently flaps, increase both values to `200m`: - -Edit the VEG resource: - -```bash -kubectl -n "${NAMESPACE}" edit veg "${VEG_NAME}" -``` - -Check the existing `spec.resources` configuration. If it is not configured, add the complete resource block below. Kube-OVN uses the entire user-provided block and does not add the default memory or ephemeral-storage values after `spec.resources` is set: - -```yaml -spec: - resources: - requests: - cpu: 200m - memory: 50Mi - limits: - cpu: 200m - memory: 50Mi - ephemeral-storage: 1Gi -``` - -If the VEG already defines custom memory or ephemeral-storage values, preserve those values and change only the CPU request and limit to `200m`. - -Wait for the generated Deployment to finish rolling out: - -```bash -kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ - --timeout=180s -``` - -### Optional: Increase the BFD intervals - -If both `minRX` and `minTX` are `100ms` and the BFD session flaps during short scheduling delays or network jitter, increase them to `1000ms`: - -Edit the VEG resource: - -```bash -kubectl -n "${NAMESPACE}" edit veg "${VEG_NAME}" -``` - -Update only `minRX` and `minTX` under `spec.bfd`. Preserve the other BFD settings: - -```yaml -spec: - bfd: - minRX: 1000 - minTX: 1000 -``` - -Wait for the generated Deployment to finish rolling out: - -```bash -kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ - --timeout=180s -``` - -With `multiplier: 5`, this change increases the approximate failure detection time from `500ms` to `5s`. If the VEG already uses `1000ms`, no change is required. - -### Apply the temporary liveness probe - -Edit the generated Deployment: - -```bash -kubectl -n "${NAMESPACE}" edit deployment "${DEPLOYMENT}" -``` - -Find the container named `bfdd` and replace only its existing `livenessProbe` with the following configuration. Preserve the other container and Deployment fields: - -```yaml -livenessProbe: - exec: - command: - - bash - - -ec - - | - output="$(bfdd-control status)" - - if grep -q '^There are 0 sessions:' <<< "${output}"; then - exit 1 - fi - initialDelaySeconds: 15 - periodSeconds: 5 - timeoutSeconds: 3 - failureThreshold: 3 - successThreshold: 1 -``` - -Wait for the rollout to finish: - -```bash -kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ - --timeout=180s -``` - -Keep the existing readiness probe unchanged. Making readiness depend on the BFD session being `Up` can prevent a new or recovering VEG Pod from being selected while the BFD session is being established. - -## Verification - -Check the BFD session and `bfdd` restart count in each VEG Pod: - -```bash -for pod in $(kubectl -n "${NAMESPACE}" get pods \ - -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ - -o jsonpath='{.items[*].metadata.name}'); do - echo "=== ${pod} ===" - kubectl -n "${NAMESPACE}" exec "${pod}" -c bfdd -- bfdd-control status - kubectl -n "${NAMESPACE}" get pod "${pod}" \ - -o jsonpath='bfdd restartCount={.status.containerStatuses[?(@.name=="bfdd")].restartCount}{"\n"}' -done -``` - -After the liveness probe detects `There are 0 sessions:` three consecutive times, the kubelet restarts `bfdd`. The existing startup probe then configures the BFD parameters and registers the peers again. Verify that a session is created and eventually reaches `Up`. - -## Notes - -- Do not apply these temporary changes to ACP 4.3.3 or later. Upgrade to a fixed release instead. -- The temporary liveness probe handles only an empty local session table. It does not repair an existing session that remains `Down`. -- If `bfdd` keeps restarting or the session remains `Down`, investigate the VPC BFD port, OVN-side BFD state, peer reachability, and the network path. -- The VEG Deployment is generated by the Kube-OVN controller. The manual probe change can be overwritten when the VEG specification or replica count changes, Kube-OVN is upgraded, or the VEG or Deployment is recreated. -- Increasing the BFD interval to `1000ms` reduces sensitivity to short interruptions but delays detection and failover for real failures. diff --git a/docs/en/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md b/docs/en/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md new file mode 100644 index 000000000..2e2f70445 --- /dev/null +++ b/docs/en/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md @@ -0,0 +1,108 @@ +--- +products: + - Alauda Container Platform +kind: + - Troubleshooting +ProductsVersion: + - 4.3.0,4.3.1,4.3.2 +id: TBD +--- + +# VPC Egress Gateway remains Ready with zero BFD sessions + +## Issue + +On Alauda Container Platform 4.3.0, 4.3.1, or 4.3.2, a VPC Egress Gateway (VEG) has BFD enabled and application Pods selected by the VEG cannot access external networks. The VEG Pod remains `Running` and `Ready`, but the `bfdd` container has no local BFD session: + +```text +There are 0 sessions: +``` + +This article applies only when `bfdd-control status` reports `There are 0 sessions:` while the VEG Pod remains healthy. A BFD session that exists but remains `Down` or frequently transitions between `Up` and `Down` is a different condition. + +## Root Cause + +The existing `bfdd` health probes run `bfdd-control status` directly. The command returns exit code `0` when it successfully communicates with the local daemon, even if the daemon reports no BFD sessions: + +| BFD state | Example output | Exit code | +| --- | --- | --- | +| A session exists | `There are 1 sessions:` | `0` | +| No session exists | `There are 0 sessions:` | `0` | + +Kubernetes exec probes evaluate only the command exit code and do not interpret its output. The kubelet therefore continues to consider `bfdd` healthy when the local session table is empty and does not restart the container to reinitialize its BFD configuration. Without a BFD session, the corresponding VEG next hop can become unavailable and interrupt application outbound traffic. + +## Resolution + +Upgrade the cluster to ACP 4.3.3 or later, which includes the fix for this issue. No manual health-probe change is required after the upgrade. + +If the cluster must remain on ACP 4.3.0, 4.3.1, or 4.3.2, use the following temporary workaround. + +Set the VEG namespace and name, and obtain the generated Deployment name: + +```bash +NAMESPACE="" +VEG_NAME="" + +DEPLOYMENT="$(kubectl -n "${NAMESPACE}" get deployment \ + -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ + -o jsonpath='{.items[0].metadata.name}')" +``` + +Edit the generated Deployment: + +```bash +kubectl -n "${NAMESPACE}" edit deployment "${DEPLOYMENT}" +``` + +Find the container named `bfdd` and replace only its existing `livenessProbe` with the following configuration. Keep the readiness probe and all other Deployment fields unchanged: + +```yaml +livenessProbe: + exec: + command: + - bash + - -ec + - | + output="$(bfdd-control status)" + + if grep -q '^There are 0 sessions:' <<< "${output}"; then + exit 1 + fi + initialDelaySeconds: 15 + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 3 + successThreshold: 1 +``` + +Wait for the Deployment rollout to finish: + +```bash +kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ + --timeout=180s +``` + +:::warning +Updating the Deployment recreates VEG Pods. For a single-replica VEG, schedule a maintenance window because application outbound traffic can be interrupted during the rollout. + +The VEG Deployment is generated by the Kube-OVN controller. This manual probe change can be overwritten when the VEG specification or replica count changes, Kube-OVN is upgraded, or the VEG or Deployment is recreated. Upgrade to ACP 4.3.3 or later instead of reapplying the workaround when an upgrade is available. +::: + +## Diagnostic Steps + +Check the local BFD session in each VEG Pod: + +```bash +for pod in $(kubectl -n "${NAMESPACE}" get pods \ + -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ + -o jsonpath='{.items[*].metadata.name}'); do + echo "=== ${pod} ===" + kubectl -n "${NAMESPACE}" exec "${pod}" -c bfdd -- bfdd-control status + kubectl -n "${NAMESPACE}" get pod "${pod}" \ + -o jsonpath='bfdd restartCount={.status.containerStatuses[?(@.name=="bfdd")].restartCount}{"\n"}' +done +``` + +When the temporary probe detects `There are 0 sessions:` three consecutive times, the kubelet restarts `bfdd`. The existing startup probe then configures the BFD parameters and registers the peer again. Confirm that a new session is created and eventually reaches `Up`. + +If `bfdd` continues to report zero sessions after repeated restarts, investigate the VPC BFD port, OVN-side BFD state, peer reachability, and the network path. Restarting the local container cannot repair an unavailable peer or a broken network path. diff --git a/docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md b/docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md deleted file mode 100644 index 98507142a..000000000 --- a/docs/zh/solutions/Mitigating_VPC_Egress_Gateway_BFD_issues.md +++ /dev/null @@ -1,195 +0,0 @@ ---- -products: - - Alauda Container Platform -kind: - - Solution -ProductsVersion: - - 4.3.0,4.3.1,4.3.2 -id: TBD -sourceSHA: 382a755423f1324ac634ff4cb04f2f9b2899fc0211bf96a53776d0f36baaeeb1 ---- - -# VPC 出口网关 BFD 异常的临时解决方案 - -## 问题现象 - -VPC 出口网关（VEG）已启用 BFD，但被 VEG 选中的业务 Pod 无法访问外部网络。此时 VEG Pod 可能仍处于 `Running` 和 `Ready` 状态，但本地 BFD 会话已经不可用。 - -问题发生时，`bfdd-control status` 可能显示本地会话表为空： - -```text -There are 0 sessions: -``` - -BFD 会话也可能频繁在 `Up` 和 `Down` 之间切换。BFD 会话不稳定时，对应的 VEG 下一跳可能变为不可用，导致业务出网流量中断。 - -## 环境 - -- Alauda Container Platform 4.3.0、4.3.1 或 4.3.2 -- 已启用 BFD 的 VPC 出口网关 -- 可以使用 `kubectl` 访问业务集群 - -有关配置 VPC 出口网关和启用 BFD 的方法，请参阅 [Alauda Container Platform 出口网关安装与使用指南](./Alauda_Container_Platform_Egress_Gateway_Installation_and_Usage_Guide.md)。 - -## 原因 - -现有 `bfdd` 健康探针直接执行 `bfdd-control status`。该命令只要能够成功连接本地守护进程，就会返回退出码 `0`，即使本地已经不存在 BFD 会话： - -| BFD 状态 | 示例输出 | 退出码 | -| --- | --- | --- | -| 存在 BFD 会话 | `There are 1 sessions:` | `0` | -| 不存在 BFD 会话 | `There are 0 sessions:` | `0` | - -Kubernetes exec 探针只判断命令退出码，不解析命令输出。因此，本地会话表为空时 Pod 仍然保持健康，kubelet 不会重启 `bfdd` 并重新初始化 BFD 配置。 - -以下配置也可能增加部分环境中 BFD 不稳定的概率： - -- ACP 4.3.0 和 4.3.1 中，`bfdd` 默认的 CPU request 和 limit 为 `50m`。CPU 限流可能延迟 BFD 报文处理。 -- 部分参照安装指南创建的 VEG 将 `minRX` 和 `minTX` 设置为 `100ms`，该间隔可能对短时调度延迟或网络抖动过于敏感。 - -这些配置是可能的影响因素，不代表所有 BFD 异常都由它们导致。 - -## 解决方案 - -将集群升级到 ACP 4.3.3 或更高版本，该版本已包含此问题的修复。升级后不需要应用下述临时修改。 - -如果集群暂时必须继续使用 ACP 4.3.0、4.3.1 或 4.3.2，请使用以下临时方案。修改 `bfdd` 存活探针，使其能够识别 `There are 0 sessions:`。连续三次检查失败后，由 kubelet 重启容器。如果环境还需要调整 CPU 或 BFD 间隔，请先修改 VEG 配置，最后再修改 Deployment 探针。 - -设置 VEG 所在的命名空间和名称，并获取控制器生成的 Deployment 名称： - -```bash -NAMESPACE="" -VEG_NAME="" - -DEPLOYMENT="$(kubectl -n "${NAMESPACE}" get deployment \ - -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ - -o jsonpath='{.items[0].metadata.name}')" -``` - -:::warning -修改 VEG 配置或 Deployment 会重建 VEG Pod。对于单副本 VEG，请安排维护窗口，因为滚动更新期间业务出网可能暂时中断。 -::: - -### 调整 `bfdd` CPU（可选） - -对于 ACP 4.3.0 和 4.3.1，或者 `bfdd` CPU request 或 limit 为 `50m` 且 BFD 会话频繁翻转的环境，可以将两个值调整为 `200m`： - -编辑 VEG 资源： - -```bash -kubectl -n "${NAMESPACE}" edit veg "${VEG_NAME}" -``` - -先检查现有 `spec.resources` 配置。如果尚未配置，请添加以下完整资源配置。设置 `spec.resources` 后，Kube-OVN 会直接使用用户提供的整个配置，不会继续补充默认的内存和临时存储值： - -```yaml -spec: - resources: - requests: - cpu: 200m - memory: 50Mi - limits: - cpu: 200m - memory: 50Mi - ephemeral-storage: 1Gi -``` - -如果 VEG 已经配置了自定义内存或临时存储值，请保留这些值，只将 CPU request 和 limit 调整为 `200m`。 - -等待控制器生成的 Deployment 完成滚动更新： - -```bash -kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ - --timeout=180s -``` - -### 调整 BFD 检测间隔（可选） - -如果 `minRX` 和 `minTX` 均为 `100ms`，并且 BFD 会话在短时调度延迟或网络抖动期间频繁翻转，可以将两个值调整为 `1000ms`： - -编辑 VEG 资源： - -```bash -kubectl -n "${NAMESPACE}" edit veg "${VEG_NAME}" -``` - -只修改 `spec.bfd` 下的 `minRX` 和 `minTX`，保留其他 BFD 配置： - -```yaml -spec: - bfd: - minRX: 1000 - minTX: 1000 -``` - -等待控制器生成的 Deployment 完成滚动更新： - -```bash -kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ - --timeout=180s -``` - -当 `multiplier: 5` 时，该调整会将近似故障检测时间从 `500ms` 延长到 `5s`。如果 VEG 已经使用 `1000ms`，则无需修改。 - -### 应用临时存活探针 - -编辑控制器生成的 Deployment： - -```bash -kubectl -n "${NAMESPACE}" edit deployment "${DEPLOYMENT}" -``` - -找到名为 `bfdd` 的容器，只将其现有 `livenessProbe` 替换为以下配置，保留容器和 Deployment 中的其他字段： - -```yaml -livenessProbe: - exec: - command: - - bash - - -ec - - | - output="$(bfdd-control status)" - - if grep -q '^There are 0 sessions:' <<< "${output}"; then - exit 1 - fi - initialDelaySeconds: 15 - periodSeconds: 5 - timeoutSeconds: 3 - failureThreshold: 3 - successThreshold: 1 -``` - -等待滚动更新完成： - -```bash -kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ - --timeout=180s -``` - -保持现有就绪探针不变。让就绪状态依赖 BFD 会话进入 `Up`，可能导致新建或正在恢复的 VEG Pod 在建立 BFD 会话期间无法被控制器选中。 - -## 验证 - -检查每个 VEG Pod 的 BFD 会话和 `bfdd` 重启次数： - -```bash -for pod in $(kubectl -n "${NAMESPACE}" get pods \ - -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ - -o jsonpath='{.items[*].metadata.name}'); do - echo "=== ${pod} ===" - kubectl -n "${NAMESPACE}" exec "${pod}" -c bfdd -- bfdd-control status - kubectl -n "${NAMESPACE}" get pod "${pod}" \ - -o jsonpath='bfdd restartCount={.status.containerStatuses[?(@.name=="bfdd")].restartCount}{"\n"}' -done -``` - -存活探针连续三次检测到 `There are 0 sessions:` 后，kubelet 会重启 `bfdd`。现有启动探针随后会重新配置 BFD 参数并注册对端。确认新的 BFD 会话已经创建并最终进入 `Up` 状态。 - -## 注意事项 - -- ACP 4.3.3 或更高版本不需要应用这些临时修改，应直接升级到已修复版本。 -- 临时存活探针只能处理本地会话表为空的情况，不能修复已经存在但持续处于 `Down` 状态的会话。 -- 如果 `bfdd` 持续重启或会话持续处于 `Down`，需要检查 VPC BFD 端口、OVN 侧 BFD 状态、对端连通性和网络路径。 -- VEG Deployment 由 Kube-OVN 控制器生成。修改 VEG spec 或副本数、升级 Kube-OVN，或者重建 VEG 或 Deployment 时，手工修改的探针可能被覆盖。 -- 将 BFD 检测间隔调整为 `1000ms` 可以降低对短时抖动的敏感度，但也会延长真实故障的发现和切换时间。 diff --git a/docs/zh/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md b/docs/zh/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md new file mode 100644 index 000000000..11c2d9756 --- /dev/null +++ b/docs/zh/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md @@ -0,0 +1,109 @@ +--- +products: + - Alauda Container Platform +kind: + - Troubleshooting +ProductsVersion: + - 4.3.0,4.3.1,4.3.2 +id: TBD +sourceSHA: 094b7ce110a0f72eeba6610815b3cf2edbbfab001620d9901a256903604f31b8 +--- + +# VPC 出口网关 BFD 会话为空但 Pod 仍为 Ready + +## 问题现象 + +在 Alauda Container Platform 4.3.0、4.3.1 或 4.3.2 中，VPC 出口网关（VEG）已启用 BFD，被 VEG 选中的业务 Pod 无法访问外部网络。此时 VEG Pod 仍处于 `Running` 和 `Ready` 状态，但 `bfdd` 容器中没有本地 BFD 会话： + +```text +There are 0 sessions: +``` + +本文仅适用于 `bfdd-control status` 显示 `There are 0 sessions:`，但 VEG Pod 仍然健康的场景。已经存在 BFD 会话但状态持续为 `Down`，或者会话频繁在 `Up` 和 `Down` 之间切换，属于其他问题。 + +## 根因 + +现有 `bfdd` 健康探针直接执行 `bfdd-control status`。该命令只要能够成功连接本地守护进程，就会返回退出码 `0`，即使本地已经不存在 BFD 会话： + +| BFD 状态 | 示例输出 | 退出码 | +| --- | --- | --- | +| 存在 BFD 会话 | `There are 1 sessions:` | `0` | +| 不存在 BFD 会话 | `There are 0 sessions:` | `0` | + +Kubernetes exec 探针只判断命令退出码，不解析命令输出。因此，本地会话表为空时，kubelet 仍然认为 `bfdd` 健康，不会重启容器并重新初始化 BFD 配置。没有 BFD 会话时，对应的 VEG 下一跳可能变为不可用，导致业务出网流量中断。 + +## 解决方案 + +将集群升级到 ACP 4.3.3 或更高版本，该版本已包含此问题的修复。升级后不需要手工修改健康探针。 + +如果集群暂时必须继续使用 ACP 4.3.0、4.3.1 或 4.3.2，请使用以下临时方案。 + +设置 VEG 所在的命名空间和名称，并获取控制器生成的 Deployment 名称： + +```bash +NAMESPACE="" +VEG_NAME="" + +DEPLOYMENT="$(kubectl -n "${NAMESPACE}" get deployment \ + -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ + -o jsonpath='{.items[0].metadata.name}')" +``` + +编辑控制器生成的 Deployment： + +```bash +kubectl -n "${NAMESPACE}" edit deployment "${DEPLOYMENT}" +``` + +找到名为 `bfdd` 的容器，只将其现有 `livenessProbe` 替换为以下配置。保持就绪探针和 Deployment 中的其他字段不变： + +```yaml +livenessProbe: + exec: + command: + - bash + - -ec + - | + output="$(bfdd-control status)" + + if grep -q '^There are 0 sessions:' <<< "${output}"; then + exit 1 + fi + initialDelaySeconds: 15 + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 3 + successThreshold: 1 +``` + +等待 Deployment 完成滚动更新： + +```bash +kubectl -n "${NAMESPACE}" rollout status deployment/"${DEPLOYMENT}" \ + --timeout=180s +``` + +:::warning +更新 Deployment 会重建 VEG Pod。对于单副本 VEG，请安排维护窗口，因为滚动更新期间业务出网可能暂时中断。 + +VEG Deployment 由 Kube-OVN 控制器生成。修改 VEG spec 或副本数、升级 Kube-OVN，或者重建 VEG 或 Deployment 时，手工修改的探针可能被覆盖。如果可以升级，请直接升级到 ACP 4.3.3 或更高版本，不要重复应用临时方案。 +::: + +## 诊断步骤 + +检查每个 VEG Pod 中的本地 BFD 会话： + +```bash +for pod in $(kubectl -n "${NAMESPACE}" get pods \ + -l "ovn.kubernetes.io/vpc-egress-gateway=${VEG_NAME}" \ + -o jsonpath='{.items[*].metadata.name}'); do + echo "=== ${pod} ===" + kubectl -n "${NAMESPACE}" exec "${pod}" -c bfdd -- bfdd-control status + kubectl -n "${NAMESPACE}" get pod "${pod}" \ + -o jsonpath='bfdd restartCount={.status.containerStatuses[?(@.name=="bfdd")].restartCount}{"\n"}' +done +``` + +临时探针连续三次检测到 `There are 0 sessions:` 后，kubelet 会重启 `bfdd`。现有启动探针随后会重新配置 BFD 参数并注册对端。确认新的 BFD 会话已经创建并最终进入 `Up` 状态。 + +如果 `bfdd` 在反复重启后仍然显示零个会话，需要检查 VPC BFD 端口、OVN 侧 BFD 状态、对端连通性和网络路径。重启本地容器无法修复对端不可用或网络路径异常。 From b90efdd3cfb6dc2a757882ff69e202b0f611482b Mon Sep 17 00:00:00 2001 From: clyi Date: Wed, 22 Jul 2026 15:16:02 +0800 Subject: [PATCH 5/6] docs: align VEG BFD recovery guidance --- ...C_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md | 4 ++-- ...C_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/en/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md b/docs/en/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md index 2e2f70445..bc1f0dc5b 100644 --- a/docs/en/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md +++ b/docs/en/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md @@ -18,7 +18,7 @@ On Alauda Container Platform 4.3.0, 4.3.1, or 4.3.2, a VPC Egress Gateway (VEG) There are 0 sessions: ``` -This article applies only when `bfdd-control status` reports `There are 0 sessions:` while the VEG Pod remains healthy. A BFD session that exists but remains `Down` or frequently transitions between `Up` and `Down` is a different condition. +This article applies only when `bfdd-control status` reports `There are 0 sessions:` while the VEG Pod remains healthy. ## Root Cause @@ -68,7 +68,7 @@ livenessProbe: if grep -q '^There are 0 sessions:' <<< "${output}"; then exit 1 fi - initialDelaySeconds: 15 + initialDelaySeconds: 1 periodSeconds: 5 timeoutSeconds: 3 failureThreshold: 3 diff --git a/docs/zh/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md b/docs/zh/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md index 11c2d9756..ad2bc8287 100644 --- a/docs/zh/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md +++ b/docs/zh/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md @@ -6,7 +6,7 @@ kind: ProductsVersion: - 4.3.0,4.3.1,4.3.2 id: TBD -sourceSHA: 094b7ce110a0f72eeba6610815b3cf2edbbfab001620d9901a256903604f31b8 +sourceSHA: ee5281812ca61cafbff02ff6aedf5c20eb2a9e93f47a35919031a922cd1db5cb --- # VPC 出口网关 BFD 会话为空但 Pod 仍为 Ready @@ -19,7 +19,7 @@ sourceSHA: 094b7ce110a0f72eeba6610815b3cf2edbbfab001620d9901a256903604f31b8 There are 0 sessions: ``` -本文仅适用于 `bfdd-control status` 显示 `There are 0 sessions:`，但 VEG Pod 仍然健康的场景。已经存在 BFD 会话但状态持续为 `Down`，或者会话频繁在 `Up` 和 `Down` 之间切换，属于其他问题。 +本文仅适用于 `bfdd-control status` 显示 `There are 0 sessions:`，但 VEG Pod 仍然健康的场景。 ## 根因 @@ -69,7 +69,7 @@ livenessProbe: if grep -q '^There are 0 sessions:' <<< "${output}"; then exit 1 fi - initialDelaySeconds: 15 + initialDelaySeconds: 1 periodSeconds: 5 timeoutSeconds: 3 failureThreshold: 3 From 02f905c3abb8d79d0b9d787f9955e7d1313740ea Mon Sep 17 00:00:00 2001 From: clyi Date: Wed, 22 Jul 2026 16:07:28 +0800 Subject: [PATCH 6/6] docs: simplify VEG BFD probe command --- ...gress_Gateway_remains_Ready_with_zero_BFD_sessions.md | 7 +------ ...gress_Gateway_remains_Ready_with_zero_BFD_sessions.md | 9 ++------- 2 files changed, 3 insertions(+), 13 deletions(-) diff --git a/docs/en/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md b/docs/en/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md index bc1f0dc5b..332b6ae4d 100644 --- a/docs/en/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md +++ b/docs/en/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md @@ -62,12 +62,7 @@ livenessProbe: command: - bash - -ec - - | - output="$(bfdd-control status)" - - if grep -q '^There are 0 sessions:' <<< "${output}"; then - exit 1 - fi + - 'output="$(bfdd-control status)"; ! grep -q "^There are 0 sessions:" <<< "${output}"' initialDelaySeconds: 1 periodSeconds: 5 timeoutSeconds: 3 diff --git a/docs/zh/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md b/docs/zh/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md index ad2bc8287..ed9c8cf4e 100644 --- a/docs/zh/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md +++ b/docs/zh/solutions/VPC_Egress_Gateway_remains_Ready_with_zero_BFD_sessions.md @@ -6,7 +6,7 @@ kind: ProductsVersion: - 4.3.0,4.3.1,4.3.2 id: TBD -sourceSHA: ee5281812ca61cafbff02ff6aedf5c20eb2a9e93f47a35919031a922cd1db5cb +sourceSHA: 876c743ff16cecb573589b3657be2ed43b666921c72c20ac0be4b41553a9eca6 --- # VPC 出口网关 BFD 会话为空但 Pod 仍为 Ready @@ -63,12 +63,7 @@ livenessProbe: command: - bash - -ec - - | - output="$(bfdd-control status)" - - if grep -q '^There are 0 sessions:' <<< "${output}"; then - exit 1 - fi + - 'output="$(bfdd-control status)"; ! grep -q "^There are 0 sessions:" <<< "${output}"' initialDelaySeconds: 1 periodSeconds: 5 timeoutSeconds: 3