Skip to content

apisnoop-backports.org

Latest commit

 

History

History
266 lines (228 loc) · 12.3 KB

apisnoop-backports.org

File metadata and controls

266 lines (228 loc) · 12.3 KB

APISnoop e2e test - user agent

Test-Grid Artifacts

dev/latest

master

https://k8s-testgrid.appspot.com/conformance-gce#GCE,%20master%20(dev) https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance/590 https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance/590/artifacts/bootstrap-e2e-master/kube-apiserver-audit.log

1.12

https://k8s-testgrid.appspot.com/conformance-gce#GCE,%20v1.12%20(dev) https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-latest-1-12/55 https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-latest-1-12/55/artifacts/bootstrap-e2e-master/kube-apiserver-audit.log

1.11

https://k8s-testgrid.appspot.com/conformance-gce#GCE,%20v1.11%20(dev) https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-latest-1-11/392 https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-latest-1-11/392/artifacts/bootstrap-e2e-master/kube-apiserver-audit.log

1.10

https://k8s-testgrid.appspot.com/conformance-gce#GCE,%20v1.10%20(dev) https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-latest-1-10/550 https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-latest-1-10/550/artifacts/bootstrap-e2e-master/kube-apiserver-audit.log

1.9

https://k8s-testgrid.appspot.com/conformance-gce#GCE,%20v1.9%20(dev) https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-latest-1-9/550 https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-latest-1-9/550/artifacts/bootstrap-e2e-master/kube-apiserver-audit.log

release/stable

1.11

https://k8s-testgrid.appspot.com/conformance-gce#GCE,%20v1.11%20(release) https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-stable-1-11/55 https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-latest-1-11/392/artifacts/bootstrap-e2e-master/kube-apiserver-audit.log

1.10

https://k8s-testgrid.appspot.com/conformance-gce#GCE,%20v1.10%20(release) https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-stable-1-10/550 https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-stable-1-10/550/artifacts/bootstrap-e2e-master/kube-apiserver-audit.log

1.9

https://k8s-testgrid.appspot.com/conformance-gce#GCE,%20v1.9%20(release) https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-stable-1-9/550 https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-stable-1-9/550/artifacts/bootstrap-e2e-master/kube-apiserver-audit.log

How do you process the audit logs?

/home/apisnoop/dev/audit-log-review/logs/

cd /home/apisnoop/dev/audit-log-review/
pip install -r requirements.txt
#cat logs/dev-1.12.log \
# cat logs/dev-1.12.log \
grep e2e.test/ logs/dev-1.12.log \
  | head -1000000000 \
  | jq .userAgent -r \
  | sort \
  | uniq
  #| jq .
# Still need some sed foo here

# What branch to grab swagger.json from
OPENAPI_REF=v1.10.7 # dev-1.10
OPENAPI_REF=v1.9.9 # dev-1.9
OPENAPI_REF=4d9bb04 # dev-1.12
OPENAPI_REF=v1.11.2 # dev-1.11
# The name of the 
K8S_SRC_REF=dev-1.11
AUDIT_LOG=./logs/$K8S_SRC_REF.log
AUDIT_CSV=/home/apisnoop/www/dist/$K8S_SRC_REF.csv
python logreview.py load-audit $AUDIT_LOG $OPENAPI_REF $K8S_SRC_REF

you can find all the options for exporting as a csv here: https://github.com/cncf/apisnoop#exporting-data

python logreview.py export-data  app-usage-endpoints $AUDIT_CSV $K8S_SRC_REF

How to make per e2e test user-agent + audit logs

Get the Source

git clone https://github.com/ii/kubernetes.git ~/k8s

Check out the correct apisnoop backport

BACKPORT=v1.9.9-apisnoop
BACKPORT=v1.10.7-apisnoop
BACKPORT=v1.11.2-apisnoop
cd ~/k8s
git checkout $BACKPORT

Build and deploy the cluster

DIND

GCE deploys

The kubetest gce process is very slow, particularly the teardown process needed before each deploy.

kubetest --build=bazel --up --provider=gce --gcp-project=ii-coop

Run the e2e test suite (directly)

This is the most direct way to run the tests, but it can still be very slow.

time bazel build //test/e2e:e2e.test
PREFIX=./bazel-bin/test/e2e
export GINKO_PARALLEL=y
# Will run 179 of 1032 specs
export KUBECONFIG=~/.kube/config
time $PREFIX/e2e.test \
  --v=2 \
  --provider=skeleton \
  --ginkgo.seed=1436380640 \
  --ginkgo.parallel.total=48 \
  --ginkgo.focus='\[Conformance\]' \
  --ginkgo.skip='\[Slow\]|\[Serial\]|\[Disruptive\]|\[Flaky\]|\[Feature:.+\]'

Retrieve the audit logs from the API Server

Varies per provider

Backporting apisnoop work for 1.9,1.10,1.11

Kubernetes 1.12 includes both the e2e test-as-userAgent and userAgent-auditLogging patches.

Version 1.11 and prior will need them applied in order to work.

1.12: this is fine

cd ~/go/src/k8s.io/kubernetes
git checkout release-1.12

Backport e2e-user-agent-test-auditing

We’ll need to backport these patches to increase coverage

cd ~/go/src/k8s.io/kubernetes
git checkout -b v1.11.2-apisnoop v1.11.2
cd ~/go/src/k8s.io/kubernetes
git checkout -b v1.10.7-apisnoop v1.10.7
cd ~/go/src/k8s.io/kubernetes
git checkout -b v1.9.9-apisnoop v1.9.9

Backport by cherry picking

# https://github.com/kubernetes/kubernetes/pull/67550/commits
git cherry-pick 229ecedac50 # send current e2e.test as user-agent
# https://github.com/kubernetes/kubernetes/pull/64812/commits
git cherry-pick d066d547cc # logging user-agent in audit
# git cherry-pick b5990b78cb # auto gen # Will not apply cleanly to 1.9
git cherry-pick a8b0ccc70c # add ut for audit useragent
# not sure I want this backported, also does not apply cleanly to 1.9
# git cherry-pick f0b1f1c2f6 # limit User-Agent max length 1024 and add ...TRUNCATED suffix

The oneliner to build, up, and test on gke

cd ~/go/src/k8s.io/kubernetes
kubetest --build=bazel --up --provider=gce --gcp-project=ii-coop

This approach to running the tests takes a very long time

I wonder if there is a way to speed this up.

cd ~/go/src/k8s.io/kubernetes
time bazel build //test/e2e:e2e.test
PREFIX=./bazel-bin/test/e2e
export GINKO_PARALLEL=y
# Will run 179 of 1032 specs
export KUBECONFIG=~/.kube/config
time $PREFIX/e2e.test \
  --ginkgo.parallel.total=48 \
  --ginkgo.focus='\[Conformance\]' \
  --ginkgo.skip='\[Slow\]|\[Serial\]|\[Disruptive\]|\[Flaky\]|\[Feature:.+\]' \
  --ginkgo.seed=1436380640 \
  --v=2 \
  --provider=skeleton

Footnotes