Rebuild the monitoring stack

[ameukam]

The monitoring stack is currently spread between 2 cloud providers with standalone deployments which make maintainers navigate between different visualizations platorms (monitoring-(eks|gke).prow.k8s.io) for the metrics produced by the CI.

We historically used to have monitoring.prow.k8s.io for this but at the time we only had GCP. we now have extended the infra to multiple cloud providers.

We should attempt to centralize all those metrics from all our CI clusters.

[ameukam]

IMHO, this should a subproject under this SIG.

/help
/sig k8s-infra
/lifecycle frozen
(I don't know how long it's will take to do this)

[k8s-ci-robot]

@ameukam:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

Details

In response to this:

IMHO, this should a subproject under this SIG.

/help
/sig k8s-infra
/lifecycle frozen
(I don't know how long it's will take to do this)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[elieserr]

I would be happy to help with this if possible

[elieserr]

I see we have in place the deployment for GKE in here and for EKS config is mostly here, I can spent a min looking at what we have and try to see what the next steps would be

In case you have an opinion on what next steps look like, please let me know

[ameukam]

Have an understanding of what we have is a good start. I don't have a strong opinion what are the next steps but I'm happy to talk about it.

[elieserr]

I think i have a better perspective of this, I was able to track what we have in https://monitoring-eks.prow.k8s.io/ and https://monitoring-gke.prow.k8s.io/ and where everything is created.

What I am now missing is where the config for https://monitoring.prow.k8s.io/ is? i see its consuming metrics from thanos-query-frontend.monitoring but i wasn't able to get where this was set up, maybe the case that the main grafana is a clickops instance? or maybe im missing something

also please correct me if im wrong but all this monitoring changes has been applied manually right?

[elieserr]

A couple of things come to mind thinking of how to move forward, we could

1. Use the existing grafana instance served in monitoring.prow.k8s.io and tune it to be able to consume the data from the EKS cluster, this will require

   • exposing the EKS prometheus instance outside of the cluster

   • make sure we have something like cluster labels in both prometheus instances so we can query metrics by clusters

2. Implement setup using something like Thanos, it will allows to aggregate many prometheus instances + a main grafana

3. Use Datadog as a monitoring solution, I saw there is some interest for this on slack from @upodroid

   • I think can be done using the existing prometheus as data sources.
   • Or we could deprecate the existing monitoring setup and go fully datadog.

I don't want to overthink anything so any early opinions are appreciated!

Also on the having a monitoring subproject I think it might be a good idea/opportunity to get this sorted out in a consistent way, im just worried about the LOE for getting stuff out of this repo. I would be open to champion this @ameukam

[ameukam]

I think i have a better perspective of this, I was able to track what we have in monitoring-eks.prow.k8s.io and monitoring-gke.prow.k8s.io and where everything is created.

What I am now missing is where the config for monitoring.prow.k8s.io is? i see its consuming metrics from thanos-query-frontend.monitoring but i wasn't able to get where this was set up, maybe the case that the main grafana is a clickops instance? or maybe im missing something

also please correct me if im wrong but all this monitoring changes has been applied manually right?

@elieser1101 You may be right. @upodroid redeployed the stack during the prow migration.

[ameukam]

A couple of things come to mind thinking of how to move forward, we could

1. Use the existing grafana instance served in `monitoring.prow.k8s.io` and tune it to be able to consume the data from the EKS cluster, this will require
   
   * exposing the EKS prometheus instance outside of the cluster
   * make sure we have something like cluster labels in both prometheus instances so we can query metrics by clusters

2. Implement setup using something like [Thanos](https://thanos.io/tip/thanos/design.md/#architecture), it will allows to aggregate many prometheus instances +  a main grafana

3. Use Datadog as a monitoring solution, I saw there is some interest for this on slack from [@upodroid](https://github.com/upodroid)
   
   * I think can be done using the [existing prometheus](https://docs.datadoghq.com/containers/kubernetes/prometheus/?tab=kubernetesadv2) as data sources.
   * Or we could deprecate the  existing monitoring setup and go [fully datadog](https://docs.datadoghq.com/containers/kubernetes/).

I don't want to overthink anything so any early opinions are appreciated!

Also on the having a monitoring subproject I think it might be a good idea/opportunity to get this sorted out in a consistent way, im just worried about the LOE for getting stuff out of this repo. I would be open to champion this @ameukam

(1) seems like a good start. I'm also interested on redeploy monitoring.prow.k8s.io in a environment (or cloud ?) that is not solely controlled by the SIG TLs. IMHO, it should help boostrap the monitoring subproject (I wanted to call it "Observability") and empower anyone willing to work on this.
Happy to support you in you want to drive this.