Skip to content

migrate monitoring stack to argocd #1515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
iamvigneshwars wants to merge 4 commits into
base: main
Choose a base branch
from
Open

migrate monitoring stack to argocd #1515

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b936886
fix(monitoring): add monitoring argocd app
iamvigneshwars Aug 6, 2026
c91b4a3
chore(apps): bump apps version from 0.5.13 to 0.5.14
iamvigneshwars Aug 6, 2026
4f1dc4e
fix(monitoring): update monitoring sealedsecrets
iamvigneshwars Aug 6, 2026
98ac486
chore(monitoring): bump monitoring version from 0.1.25 to 0.1.26
iamvigneshwars Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion charts/apps/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@ apiVersion: v2
name: apps
description: An argocd app to deploy apps inside the virtual cluster
type: application
version: 0.5.13
version: 0.5.14
3 changes: 3 additions & 0 deletions charts/apps/dev-values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,9 @@ dashboard:
extraValueFiles:
- dev-values.yaml

monitoring:
enabled: false

identityMapper:
enabled: false
targetRevision: HEAD
Expand Down
3 changes: 3 additions & 0 deletions charts/apps/staging-values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,9 @@ otel:
extraValuesFiles: []
valuesObject: {}

monitoring:
enabled: false

@TBThomas56 TBThomas56 Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason why it’s switched off in staging?

@iamvigneshwars iamvigneshwars Aug 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SInce we don't have a sealedsecret controller inside the vcluster in pollux, thought it might be better to wait for your openbao migration

@davehadley davehadley Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SealedSecrets controller should also be disabled in production. See:

workflows/charts/apps/values.yaml

Line 119 in d858c70

enabled: false

The current strategy is to seal secrets in the host cluster,
deploy them via the secrets charts:
https://github.com/DiamondLightSource/workflows/tree/main/charts/workflows-cluster/charts/secrets
Copy them to the vCluster via vCluster syncing:

workflows/charts/workflows-cluster/values.yaml

Line 142 in d858c70

"/letsencrypt-argo-workflows-workflows-diamond-ac-uk": "workflows/workflows-tls-cert"


kueue:
enabled: true
targetRevision: HEAD
Expand Down
32 changes: 32 additions & 0 deletions charts/apps/templates/monitoring-application.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
{{- if .Values.monitoring.enabled }}
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: monitoring
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "0"
spec:
destination:
namespace: monitoring
server: {{ .Values.destination.server }}
project: default
source:
repoURL: https://github.com/DiamondLightSource/workflows.git
path: charts/monitoring
targetRevision: {{ .Values.monitoring.targetRevision }}
helm:
valueFiles:
- values.yaml
{{- if .Values.monitoring.extraValueFiles }}
{{- .Values.monitoring.extraValueFiles | toYaml | nindent 8 }}
{{- end }}
{{- if .Values.monitoring.valuesObject }}
valuesObject:
{{- .Values.monitoring.valuesObject | toYaml | nindent 8 }}
{{- end }}
syncPolicy:
automated:
prune: true
selfHeal: true
{{- end }}
6 changes: 6 additions & 0 deletions charts/apps/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,12 @@ otel:
extraValuesFiles: []
valuesObject: {}

monitoring:
enabled: true
targetRevision: HEAD
extraValuesFiles: []
valuesObject: {}

kueue:
enabled: true
targetRevision: HEAD
Expand Down
2 changes: 1 addition & 1 deletion charts/monitoring/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v2
name: monitoring
description: A monitoring stack for the workflows deployment
type: application
version: 0.1.25
version: 0.1.26
dependencies:
- name: grafana
repository: https://grafana.github.io/helm-charts
Expand Down
2 changes: 2 additions & 0 deletions charts/monitoring/templates/grafana-dashboards.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,7 @@ kind: ConfigMap
metadata:
name: grafana-dashboards
namespace: {{ .Release.Namespace }}
annotations:
argocd.argoproj.io/sync-options: ServerSideApply=true
data:
{{ (.Files.Glob "grafana-dashboards/*.json").AsConfig | nindent 2 }}
8 changes: 4 additions & 4 deletions charts/monitoring/templates/grafana-oauth-secret.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@ apiVersion: bitnami.com/v1alpha1
kind: SealedSecret

@davehadley davehadley Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above comment.

All sealed secrets have been migrated to:
https://github.com/DiamondLightSource/workflows/tree/main/charts/workflows-cluster/charts/secrets
sealed in the host cluster, and copied to the vCluster destination.

This eliminates the need for us to run and manage our own SealedSecrets controller.

metadata:
name: grafana-oauth
namespace: workflows
namespace: monitoring

@TBThomas56 TBThomas56 Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has to be workflows (referring to the namespace the vcluster is installed in). It is then mapped in both the values.yaml and staging-values.yaml to its destination namespace within the vcluster

@iamvigneshwars iamvigneshwars Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The monitoring chart is rendered inside the vcluster, with monitoring namespace as the destination. So, this template must be in the monitoring namespace

spec:
encryptedData:
client-id: AgB6cWGK5n9LSkX29hVE82ERqV8B3+PU5va4O4s4Dbmk6/IRiJ0P8dn+Xkc0D57dx71HkhkZ4I0mQKj/T4ZfSE0NK7yW4OXCobQMQC3532MQHlBU6gweaAcak89fKgACLM9xCe23JKr9IlSROaNTHEhrpFktoq89PRh8u4zVo0wFsUNpxqqc/LZo/ICC/2CGht3AMxKlcDV66f1XE9GEYMNEHV5qXxq4dCrzBXnTgJZ5aTLUpW0xHUfYvPSmZ2kynGS2UTSrArbp++nphO9hzKg6Id6hPuqeptXO907gDeR1rykEXZkFmVuROWXMdGna/4Cgd5whAktkt9LeURvwa5zQ3UA6y+P2bCHPtKMN4760k+1GTiC3fP+YJkRIjgGTF+TJVKdSzJ6fZ5794IVZ/apdlrapxED7+v8uGRdW+RymrHUMZ26tj0FiSiW0wjvVq4v4GgpxTqmnHF+0U/Rv5402FkASh5/0G52gOAKX7Y/Cur2qozbxy/RVstpiV7BZJb3+8+0PdyUr5znY1tHCJjHeF6T8f5p5BQaMARkIFCkpxW9Q3Lt5iyGLWDsRevuw+nGuQTBfXbcTA6ThMIK1g7qNMH60Roj+robO4SKdWXedB+qEdLMerVOlmagC/EeF9gGZ2NCjTfu3blNjkLdhCV25fJAxhyQC/OwBug585nHA8Kz9jowsFND4+xn0jMNm1nq586687rbc3F4J1fMNQ/FOkg==
client-secret: AgB+r2lmnpgSCCnPVfLa24mC1u+nut4WXUVfVWgcDVZQKzVIniZWWa6HPxO+B/V1apcZ60XuyFZGCaEXoFRULRSD/BUOuWvmMnKjU5V15UwOuOZUS39yP0smHRZ2FRfJ/fMbARW6iBt+hfnGZzt2FlIVBiXnQNcgAK9cI0pz17jDeNThT/wvsjNfRkx+UiBzhoa9eukSWMD84+Yi/gllUgDMQcEo2Ej+Ndc3/unlPxSlsKkHGCFpBuJiRqrhfJpJ272hrMgj0ZMY6FgWvg68ZixON7FAO4djuRpo12YWFDWGmKO8UoGEWeQaHQDDioB3I2wizynLItGFF/VrmoTgKuWknET0jktB27wUGT/fsvni5qNKrNvWfBUvMfy9voicWWmDR2ZkH9StUUj8QqaWBpv13qZBZ0mk1HM1CheUehfb0U5rHj9kMUJnHwkrvoPSWuQO94TG+U3Sg6V+yFdnQk+jfCrt+TroD5FHYppx564RiEC+qlfRlVOaZz6L1QFeejbN7XK+uUC4UH0ldKHyzw4GZdpkiTOfShOdKi9Z4wlZNCRrb2t51ghtidXlN3Cfk6ZTXSkNfYxyJ5+c7YHJvMhvZkhcJ6yg2cLXARp3eBNJhATWVcbLNFCL+KuR2oTpH6DhjUhAUSrhZjSu/L8rK1DtgNRYHjfI1Nb8xjdny5RdSETtbutHbkevTY8UuQjY3BFaTCON7w+sGEz0v0PXfBiCD6OKHWQ1LRS+UkXVcWaFiA==
client-id: AgCa9cKxRR8lXcin4sk6ww6tQQQQe1OaBD+7+k1GSAj0jxnwfPy+vBvlbwQFrC8nvf+kcytRC6iz9GC9cn6mwCsd/w/OeW7tgbVdEZEIOn6ZCE+4fGDXa40G/5sJDvyCacUrWEvYpI6fWvvOBCT82PEX251INsFf5kfJuk5QYVWeUVfuR8tZNS1cUhVSqwcmg2KO/jSueqW5v6Nesp2b5AApujro5DPbUbyXEmUqSHQZNVRWnxDLln4vZ8vbyBiCmOlHtY05Gbc4UEM+k9N9nD0Dz8Mo07IxnwGwm55vQg1hZktHdWfvE60SC18TDC/cA4Q30nM9zgRSw9T0gqinXD+7vA16v1JwNGKtaUVzi85lmaBw0f8I+LmOg3T6VWNj3k7cCSgnAHsYY490IzdlR9X4hA0dZ/YC2mBu+E+iRzN8RquigtQpQALGwja7AP6AGma3WPVPfNCaSG73KYiPFn6072bNOmsdkamyFWg3XnfhoI7bGKgAjjuOpevoCIcrgFpw37/VPZYnvWdMs3uq18smBzAcUSCYPbfHjF8BEEAzBl6EkuFoVHHRV3sH3s5Lx2IdLfel+UEW1wULj1ymBqoywkCvuc05zAIKmay36gzPUlzuvxCIcpZXrvb20S3fpIDS45d89iYvTPR34X+XXhNz2w8YOYDeMVhide05gV6mJ0rjVN6aUDWwO91fdMrZdXvT74yM5DiW0M2zz1DYsfJafw==
client-secret: AgCwBoPid8wtKu4SVESPAB42MQSk8A7XPxM4zLCPAiW4eAgYzUPMlApqe66fO7SlvH9lqkQysR72aVRjmPTRLdcPdmvmuXETUBHr8hre63xzNCrqSQvBn11EfIjEuVjb3kUCaqiEt2JJjNU6VZY8uec2OZn+8Nd2kQbpSjVwtXGuQuJ5PEhg+5XSG0sPOmqVIkftzIm24x+97AAKp5G0IvC4wXhQMkLjTHOKRgNbMqK5mBhO3neI5wMiw/AzaOXuxy2DW+XfjMbj60a7ojQ5iePaR0Z7UJjscWECv6awJAjF0WUEvOx7yNrTM8MQxEtCeB/EdvHvdlT/CbG1dmu3ZM1G5OnZBjAFa38CNuOgxpukp785DLfA6vXJP/N6iIA/CSbIJUf+98i4eiR+fBOy7M62MJZKcYl47vcASiQuNnsGBrKisuGg/Mjcq0hBvegTeu83f2cRZvKzXhq3XOBvqYZ9xJAkkxGqG0vymutqZ25V7g+WT9ZyXxSYFd8xXuo5/z5CeSvQxCXRtJM56J+dkLZExZlIlUEYc3OeOviRwABP87A4LTRTxW5/ayykFRSw6GIuQTKf7xc+ePBmE7afmPCurlz7kxXa6c5q26qNHkUs00B30geRmewkHvQIqzA3WwUJZ1fJs1I+fQsTv2Cax3y5XkHgVU2XcDTGGSNwdgLt9/x4r0oatCx9Bidboc33u9c8zmOcJMMd2W+yc3+0LUANX/8xkvrDUuvzVCL7XAqgKg==
template:
metadata:
name: grafana-oauth
namespace: workflows
namespace: monitoring
type: Opaque
{{- else if eq .Values.cluster "pollux" }}
apiVersion: bitnami.com/v1alpha1
Expand Down
2 changes: 1 addition & 1 deletion charts/monitoring/templates/prometheus-s3-sealedsecret.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ metadata:
namespace: {{ .Release.Namespace }}
spec:
encryptedData:
objstore.yml: AgANzoxM2DcAtAMIc3MhaeTfAvuJ3zaD7JC6Ca7BjeuLfLc6pUa/YdcvE2vpL5iFQxawqhH+rcy7omFbrn9bz1dd6RFdoazobtIDlsTuWm64jMk4kB8icScJe3l7koD1RD7pyEhiPNzpk4hzhv6I90/h/l5yWClzVfhLY+Pn4ovAceVkH8N7X185vbIdISvouFqkMa+BuZ/8j9FsFA4l1gvq1tQL8M8LwQm3vOhUOKk5z84j7UQ8uyk0n+AG9kdeebUSUlPpubiY8LcM2R0P3dp/Z7HOpcwMvCGc5isWQzXd/a6/w75oGBlasZUTf9s5x0W16jrSCgNaQcJ2Pf4SycBO5H3NZDCLPh3EyKyVeQ1suhhaphDzzbfUqUVipNRoaX99vMwlzFg2st5jjNmK17YTqhRcBR9uXbS3uIJcjDzaQzMpA2sSbVAFZlqDkdK7qsleOA+IWjuoIGmqmSMKeldyStkNgOlKCqIBiydMh/kk6u8h6iXkn7WZphrzS2GuZ6dxxUBtN/TaEJ53slgvSrHMXMEmtyX3UI7P2S+exL3rdZtEMSqDSswWHudBxXOP/qsLw9VlBhWIk6jZGmo32I2t1GkKLJFFHJSGWejbycE/+Qh/UOCX0vr3qmZytyAzOXzgxT1T9gd/n7BX5aak5O/M5uO+x76s5umb9Vdzc8r62UtQU9nw7KpzG9idBRvgEHI5bA2qZlpWiQIPEFGxdnKJO1yl8zmMwN8/SfoRxK/YAbkQsKYuV3BXPJUkI/+l1PpQbl87pnVpBvYli88dfjpJZtmEs8uHpj5n2qpnHp3q+cDC9hQv7JsA8ydq1kj6EkXee24yy9bk4RWzcYZQNtiwpo/CqfeUVx2CsH0ggnL6oilUMjb+vpE278+cTHyOutDkPbtdQutHMHE18ct/pHCb0Y5QBqX+TnN6J7AF+37utwRUW3+9yw==
objstore.yml: AgCQupeZ9rnYPqD1Q5czufet/qy+s3BBRI33SjMyFbvATLJZzgddNZU3zL8MafJdDFbTuei5amq6FrJki768IUgUtyLtUx4gcJsG2LuPJ25WD17XxI9EnfPLUmP8dPGmx4b467I4miYP/I9GKd2W+t2MEiTe4BPAQluvZlcy4wWXWrux2O8woxiw1flkLLvFp6r1ZJQO/LtK17iBABMLkatR3u8pBq4iuYbnHpmmT2DKNCew/PlyrgVbtJ4Fj0gcTaG2f4S3CcjxJhobYfWDK/O9NOrfuTLiWv60WozEebLzb2IGkJ6O/KLBkNwAVFAYbm75wOW6DvRCx/ZTx6DHLqT7mdbkY8GqSimQ/hqDTX1P2eW1ZzHJYRherF6IdWYVWeHMVz9J7o/XmbP/dk0iDSHE4JcuxrWp72j+Cw9cIo8YyjdKhbFhUQ66k1aoPHxgHwX1vWOsOeurtVQICsIFKwlQpiCyXk8YD49uFFnttOjcE1OBWUI/JxjPk3H0uRr6fSB0eDwDuXwy8DLGEvlqfJs9kaz7tW0/ikBgwnC6R6cj0vFY7HYYWL+BTOiCHzMkWVdrkDi1Yy0hh1GngBbgwhpeEvHPYdFiaWRK4FAXiTH8Jx3nap7W2kKxg4pEa4u7N5iFUd8TrxUSmLFUu6Sd9fItnqpU0i4jp4LpOqbhnT5wpaJP7J41GiflZCDZCK61MVh67yDxU3rpDfamYL5l+ULPVAPtAjRD5rWkVnxI7N8XrQvXZqOCs4CmVuHoDl+9mFvKpBEe1dXRupnPKFu4fkkbxbrhv5vwRvtsP1++64olgNM3fKe7rc+ZquBWBHzN80cMBzbFWYSQZl0BXwD4gMX/TKmuBfOkRA7PdjQ+LYNLAC7pkry/HXDYdn8snOYmh5s0+6xY6EXMCcUZQfP0C4gu+gbTbvoxvLETGs4X/vAX/kGMaVCpcw==
template:
metadata:
annotations:
Expand Down
8 changes: 4 additions & 4 deletions charts/monitoring/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,12 +143,12 @@ thanos:
replicaCount: 1
persistence:
enabled: false
retention:
resolutionRaw: 30d
resolution5m: 30d
resolution1h: 1y
extraArgs:
- --retention.resolution-raw=30d
- --retention.resolution-5m=30d
- --retention.resolution-1h=1y
- --consistency-delay=30m
- --wait

@davehadley davehadley Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is removal of --wait intentional?

@iamvigneshwars iamvigneshwars Aug 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--wait is always passed by default, so it rejects repeated flags causing crash loop.

https://github.com/thanos-community/helm-charts/blob/thanos-0.27.0/charts/thanos/values.yaml#L843-L849

resources:
requests:
cpu: "1"
Expand Down
Loading