Refactor GKE to Support 3-tier Nginx architecture with Upstream nodepool by jimmycgz · Pull Request #6458 · GoogleCloudPlatform/PerfKitBenchmarker

jimmycgz · 2026-02-10T18:51:47Z

Description

This PR refactors the kubernetes_nginx benchmark to implement a true 3-tier architecture (Client -> Nginx Proxy -> Upstream Backend), matching the standard GCE nginx_benchmark topology. It replaces generic vm_util calls with robust kubernetes_commands for better resource management and reliability.

Key Changes

3-Tier Topology: Now deploys:
- Client Tier: wrk2 load generator (external VM or pod).
- Proxy Tier: Nginx Reverse Proxy (LoadBalancer Service).
- Upstream Tier: Nginx Backend (ClusterIP Service).
Native Kubernetes Logic: Switched to kubernetes_commands for applying manifests, creating ConfigMaps (nginx-configs), and waiting for resources.
Dynamic Deployment: Added nginx_proxy.yaml.j2 and nginx_upstream.yaml.j2 templates for flexible configuration.
Reliability Fixes:
- Improved connectivity check to probe /random_content instead of / (resolving 403 errors).
- Fixed LoadBalancer IP retrieval with retry logic.

Supported Platforms & Limitations

GKE: Verified (v1.30+). Uses IP-based LoadBalancers.
AKS: Theoretical Support (uses IP-based LoadBalancers).
Minikube / Kind: Theoretical Support.
AWS EKS: Limitation. Currently unsupported due to dependency on GetLoadBalancerIP (expects IP, AWS returns Hostname). Future work required.

Test Commands

Example 1: HTTP Baseline (Default)

./pkb.py --benchmarks=kubernetes_nginx --cloud=GCP --zone=us-central1-a \
  --project=$PROJECT_ID --run_uri=test_http \
  --nginx_server_machine_type=c4-standard-8 \
  --config_override=kubernetes_nginx.vm_groups.clients.vm_spec.GCP.machine_type=c4-standard-32 \
  --config_override=kubernetes_nginx.container_cluster.nodepools.upstream.vm_spec.GCP.machine_type=c4-standard-16 \
  --config_override=kubernetes_nginx.container_cluster.nodepools.upstream.vm_count=2 \
  --nginx_content_size=1024 \
  --nginx_use_ssl=False

Example 2: HTTPS Validation

./pkb.py --benchmarks=kubernetes_nginx --cloud=GCP --zone=us-central1-a \
  --project=$PROJECT_ID --run_uri=test_https \
  --nginx_server_machine_type=c4-standard-8 \
  --config_override=kubernetes_nginx.vm_groups.clients.vm_spec.GCP.machine_type=c4-standard-32 \
  --config_override=kubernetes_nginx.container_cluster.nodepools.upstream.vm_spec.GCP.machine_type=c4-standard-16 \
  --config_override=kubernetes_nginx.container_cluster.nodepools.upstream.vm_count=2 \
  --nginx_content_size=1024 \
  --nginx_use_ssl=True

google-cla · 2026-02-10T18:51:59Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

jimmy-cui-sada · 2026-02-10T19:58:23Z

Removed the tmp helper container_service.py

…mmands - Refactored benchmark to use Client -> Proxy -> Upstream architecture - Migrated from generic vm_util to kubernetes_commands for native K8s support - Added nginx_proxy.yaml.j2 and nginx_upstream.yaml.j2 templates - Fixed connectivity check to target /random_content (avoids 403 Forbidden) - Added new flags for upstream machine type and count configuration

hankfreund · 2026-03-05T19:19:14Z

perfkitbenchmarker/linux_benchmarks/kubernetes_nginx_benchmark.py

+    (Client -> Nginx Proxy -> Upstream Backend) on Kubernetes.
  container_specs:
    kubernetes_nginx:
      image: k8s_nginx


Can you update to use the official nginx image instead of the ubuntu-based one that's currently defined? Please pin it to a specific version (current latest is fine).

changed per our discussion

hankfreund · 2026-03-05T19:19:52Z

perfkitbenchmarker/linux_benchmarks/kubernetes_nginx_benchmark.py

Please remove this clients section since it's unused.

nice catch, removed

hankfreund · 2026-03-25T14:32:17Z

perfkitbenchmarker/data/docker/k8s_nginx/Dockerfile.bak.20260318

This file should probably be deleted?

mauriciopoppe · 2026-03-25T15:01:32Z

perfkitbenchmarker/linux_benchmarks/kubernetes_nginx_benchmark.py

-  """Prepares a cluster to run the Nginx benchmark."""
+  """Prepares the GKE cluster with proxy and upstream nginx deployments."""
+  # 1. Create ConfigMap with merged proxy + upstream configs
  with _CreateNginxConfigMapDir() as nginx_config_map_dirname:


Could you please log the contents of the computed ConfigMap? Are the contents part of the final artifact stored after the test runs? (maybe data.ResourcePath(...) is automatically kept after the test runs)

mauriciopoppe · 2026-03-25T15:04:05Z

perfkitbenchmarker/linux_benchmarks/kubernetes_nginx_benchmark.py

 )

 BENCHMARK_NAME = 'kubernetes_nginx'
+NGINX_IMAGE = 'nginx:1.29.6'


Is this const needed here or should it be in container_specs.kubernetes_nginx.image?

mauriciopoppe · 2026-03-25T15:07:54Z

perfkitbenchmarker/linux_benchmarks/kubernetes_nginx_benchmark.py

            machine_type: Standard_D4s_v5
-      clients:
-        vm_count: 1
+      upstream:


Looking at the GCE test I see 3 groups: clients, server, upstream_servers.

In this setup I see nginx (I guess it's the server) and upstream (I guess upstream_servers) and the vm_groups.clients for the client. I believe that in this setup the client is running outside the GKE clusters, cc @hankfreund to validate this setup.

Yeah, your understanding is correct. And we decided to keep the client VMs outside of the cluster, instead of having the traffic entirely within it.

Good, thanks.

mauriciopoppe · 2026-03-25T15:13:48Z

perfkitbenchmarker/linux_benchmarks/kubernetes_nginx_benchmark.py

-  """Run a benchmark against the Nginx server."""
-  return nginx_benchmark.Run(benchmark_spec)
+  if FLAGS.nginx_throttle:
+    return nginx_benchmark._RunMultiClient(


Great work, good to see that we're reusing functions from https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/908706425bef535da7a3ff1708701ab56d6c6952/perfkitbenchmarker/linux_benchmarks/nginx_benchmark.py

jellyfishcake · 2026-03-25T18:28:15Z

perfkitbenchmarker/linux_benchmarks/kubernetes_nginx_benchmark.py

+  hostip = benchmark_spec.nginx_endpoint_ip
+  hoststr = (
+      f'[{hostip}]'
+      if isinstance(ipaddress.ip_address(hostip), ipaddress.IPv6Address)


nit; I don't remember the status of ipv6 support on competitor clouds.

jellyfishcake · 2026-03-25T18:39:16Z

perfkitbenchmarker/linux_benchmarks/kubernetes_nginx_benchmark.py

    nodepools:
      nginx:
-        vm_count: 3
+        vm_count: 1


nit: I don't have the context on why this was 3 in the first place. 1 sounds more correct.

jellyfishcake · 2026-03-25T18:43:01Z

perfkitbenchmarker/linux_benchmarks/kubernetes_nginx_benchmark.py

-      clients:
-        vm_count: 1
+      upstream:
+        vm_count: 2


nit: In the original nginx benchmark, we had 6 upstream servers. Is 2 sufficient to load the nginx servers?

jellyfishcake · 2026-03-25T18:45:59Z

perfkitbenchmarker/linux_benchmarks/kubernetes_nginx_benchmark.py

+    except errors.VirtualMachine.RemoteCommandError:
+      pass
+    logging.info('Still waiting for connectivity...')
+    time.sleep(10)


instead of sleep, prefer retrying on failures.

jimmycgz marked this pull request as draft February 10, 2026 19:00

jimmycgz force-pushed the upstream-pr-nginx-3tier branch from 7773c41 to 30b076e Compare February 10, 2026 19:02

jimmycgz changed the title ~~GKE Support 3-tier Nginx architecture with Upstream nodepool~~ Refactor GKE to Support 3-tier Nginx architecture with Upstream nodepool Feb 10, 2026

jimmycgz force-pushed the upstream-pr-nginx-3tier branch from 30b076e to 36298bc Compare February 10, 2026 19:52

jimmycgz force-pushed the upstream-pr-nginx-3tier branch from 36298bc to efa2978 Compare February 13, 2026 01:53

jimmycgz marked this pull request as ready for review February 27, 2026 20:24

hankfreund reviewed Mar 5, 2026

View reviewed changes

resolve PR, change nginx image

dc1dddc

hankfreund reviewed Mar 25, 2026

View reviewed changes

perfkitbenchmarker/data/docker/k8s_nginx/Dockerfile.bak.20260318 Outdated

Copy link

Member

hankfreund Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file should probably be deleted?

chore: remove unused Dockerfile backup from PR

5f0d389

mauriciopoppe reviewed Mar 25, 2026

View reviewed changes

jellyfishcake reviewed Mar 25, 2026

View reviewed changes

Conversation

jimmycgz commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Key Changes

Supported Platforms & Limitations

Test Commands

Example 1: HTTP Baseline (Default)

Example 2: HTTPS Validation

Uh oh!

google-cla bot commented Feb 10, 2026

Uh oh!

jimmy-cui-sada commented Feb 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jimmycgz commented Feb 10, 2026 •

edited

Loading