Grafana Dashboard sample

[awoisoak]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.13.0

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

0) Have Prometheus and Grafana deployed in the cluster
1) Enable `metrics:` in ARC controller.
2) Enable `listenerMetrics:` in ARC listeners and add `actions_github_com_scale_set_namespace` label to `gha_job_startup_duration_seconds` metric.
3) Setup Podmonitors so Prometheus can scrape the ARC metrics.

Describe the bug

The Panels relaying on actions_github_com_scale_set_namespace (Startup Duration, Job Execution & Running Jobs) are not working

Cannot read properties of undefined (reading 'config')

The issue seems to be that actions_github_com_scale_set_namespace is empty.
Replacing actions_github_com_scale_set_namespace label with namespace seems to solve the issue

Describe the expected behavior

Assuming I’m not doing anything wrong on my end, and understanding that the Grafana dashboard itself isn’t officially supported, it would be nice if the current sample worked out of the box.

Additional Context

Sample of gha_job_startup_duration_seconds_bucket metric I'm seeing:

gha_job_startup_duration_seconds_bucket{actions_github_com_scale_set_name="",actions_github_com_scale_set_namespace="",enterprise="",event_name="pull_request",job_name="Build, Test and Deploy / Android",job_workflow_name="build-test-deploy",job_workflow_ref="MY_COMPANY/MY_REPO/.github/workflows/build-test-deploy.yaml@refs/pull/1165/merge",job_workflow_target="pull/1165",organization="MY_COMPANY",repository="MY_REPO",le="0.1"} 0


The metrics I got setup in the lister:

listenerMetrics:
  counters:
    gha_started_jobs_total:
      labels:
        ["repository", "organization", "enterprise", "job_name", "event_name", "job_workflow_ref", "job_workflow_name", "job_workflow_target"]
    gha_completed_jobs_total:
      labels:
        [
          "repository",
          "organization",
          "enterprise",
          "job_name",
          "event_name",
          "job_result",
          "job_workflow_ref",
          "job_workflow_name",
          "job_workflow_target",
        ]
  gauges:
    gha_assigned_jobs:
      labels: ["name", "namespace", "repository", "organization", "enterprise", "actions_github_com_scale_set_name", "actions_github_com_scale_set_namespace"]
    gha_running_jobs:
      labels: ["name", "namespace", "repository", "organization", "enterprise", "actions_github_com_scale_set_name", "actions_github_com_scale_set_namespace"]
    gha_registered_runners:
      labels: ["name", "namespace", "repository", "organization", "enterprise", "actions_github_com_scale_set_name", "actions_github_com_scale_set_namespace"]
    gha_busy_runners:
      labels: ["name", "namespace", "repository", "organization", "enterprise", "actions_github_com_scale_set_name", "actions_github_com_scale_set_namespace"]
    gha_min_runners:
      labels: ["name", "namespace", "repository", "organization", "enterprise", "actions_github_com_scale_set_name", "actions_github_com_scale_set_namespace"]
    gha_max_runners:
      labels: ["name", "namespace", "repository", "organization", "enterprise", "actions_github_com_scale_set_name", "actions_github_com_scale_set_namespace"]
    gha_desired_runners:
      labels: ["name", "namespace", "repository", "organization", "enterprise", "actions_github_com_scale_set_name", "actions_github_com_scale_set_namespace"]
    gha_idle_runners:
      labels: ["name", "namespace", "repository", "organization", "enterprise", "actions_github_com_scale_set_name", "actions_github_com_scale_set_namespace"]
  histograms:
    gha_job_startup_duration_seconds:
      labels:
        ["repository", "organization", "enterprise", "job_name", "event_name","job_workflow_ref", "job_workflow_name", "job_workflow_target", "actions_github_com_scale_set_name", "actions_github_com_scale_set_namespace"]
      buckets:
        [
          0.01,
          0.05,
          0.1,
          0.5,
          1.0,
          2.0,
          3.0,
          4.0,
          5.0,
          6.0,
          7.0,
          8.0,
          9.0,
          10.0,
          12.0,
          15.0,
          18.0,
          20.0,
          25.0,
          30.0,
          40.0,
          50.0,
          60.0,
          70.0,
          80.0,
          90.0,
          100.0,
          110.0,
          120.0,
          150.0,
          180.0,
          210.0,
          240.0,
          300.0,
          360.0,
          420.0,
          480.0,
          540.0,
          600.0,
          900.0,
          1200.0,
          1800.0,
          2400.0,
          3000.0,
          3600.0,
        ]
    gha_job_execution_duration_seconds:
      labels:
        [
          "repository",
          "organization",
          "enterprise",
          "job_name",
          "event_name",
          "job_result",
          "job_workflow_ref",
          "job_workflow_name",
          "job_workflow_target"
        ]
      buckets:
        [
          0.01,
          0.05,
          0.1,
          0.5,
          1.0,
          2.0,
          3.0,
          4.0,
          5.0,
          6.0,
          7.0,
          8.0,
          9.0,
          10.0,
          12.0,
          15.0,
          18.0,
          20.0,
          25.0,
          30.0,
          40.0,
          50.0,
          60.0,
          70.0,
          80.0,
          90.0,
          100.0,
          110.0,
          120.0,
          150.0,
          180.0,
          210.0,
          240.0,
          300.0,
          360.0,
          420.0,
          480.0,
          540.0,
          600.0,
          900.0,
          1200.0,
          1800.0,
          2400.0,
          3000.0,
          3600.0,
        ]

Controller Logs

N/A

Runner Pod Logs

N/A

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[kenmuse]

@awoisoak - You may find this discussion helpful. Typically, the labels you're looking for are generated by Prometheus through the scrape config. Prometheus can use the pod labels to generate additional labels for the metrics data, turning this:

Labels:
  actions.github.com/scale-set-name=arc-runner-set
  actions.github.com/scale-set-namespace=arc-runners  
  ...

into

 __meta_kubernetes_pod_label_actions_github_com_scale_set_name=arc-runner-set
 __meta_kubernetes_pod_label_actions_github_com_scale_set_namespace=arc-runners

A common scrape configuration is to alter these labels, removing the prefix __meta_kubernetes_pod_label_. A configuration like this, for example, will do that (scoped to the namespace arc-systems):

scrape_configs:
  - job_name: arc-metrics
    honor_labels: true
    kubernetes_sd_configs:
      - role: pod
         namespaces:
            names:
              - arc-systems
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)

This is how actions_github_com_scale_set_namespace is created. It's not directly coming from the ARC configuration. There are two quick potential fixes with the current chart:

• Update your scrape config to relabel the scraped pod labels; or,
• Replace actions_github_com_scale_set_namespace (and similar) in your copy of the chart with the full, raw value __meta_kubernetes_pod_label_actions_github_com_scale_set_namespace.

Both of these assume that your existing scrape configuration is already capturing the pod's labels.

@nikola-jokic - we should discuss the best way to approach this to minimize confusion.