Skip to content

collector: reduce filesystem and textfile allocations#3631

Open
kevinburke wants to merge 1 commit intoprometheus:masterfrom
kevinburke:perf-bench-collector-allocs
Open

collector: reduce filesystem and textfile allocations#3631
kevinburke wants to merge 1 commit intoprometheus:masterfrom
kevinburke:perf-bench-collector-allocs

Conversation

@kevinburke
Copy link
Copy Markdown

@kevinburke kevinburke commented Apr 21, 2026

Store filesystem read-only state during mount parsing instead of serializing and reparsing mount options on every scrape. Rework textfile metric family conversion to build one desc per family and normalize label sets by index.

Benchstat:
goos: linux
goarch: amd64
pkg: github.com/prometheus/node_exporter/collector 
cpu: AMD Ryzen 9 8945HS w/ Radeon 780M Graphics

                              │ /tmp/base.txt │          /tmp/current.txt           │
                              │    sec/op     │   sec/op     vs base                │
ParseFilesystemLabels-8          219.78µ ± 0%   98.89µ ± 1%  -55.01% (p=0.000 n=20)
TextfileConvertMetricFamily-8     466.0µ ± 1%   223.5µ ± 1%  -52.04% (p=0.000 n=20)
geomean                           320.0µ        148.7µ       -53.55%

                              │ /tmp/base.txt │           /tmp/current.txt           │
                              │     B/op      │     B/op      vs base                │
ParseFilesystemLabels-8         191.29Ki ± 0%   99.28Ki ± 0%  -48.10% (p=0.000 n=20)
TextfileConvertMetricFamily-8    472.0Ki ± 0%   292.8Ki ± 0%  -37.95% (p=0.000 n=20)
geomean                          300.5Ki        170.5Ki       -43.25%

                              │ /tmp/base.txt │          /tmp/current.txt           │
                              │   allocs/op   │  allocs/op   vs base                │
ParseFilesystemLabels-8           4.169k ± 0%   1.097k ± 0%  -73.69% (p=0.000 n=20)
TextfileConvertMetricFamily-8    13.054k ± 0%   7.955k ± 0%  -39.06% (p=0.000 n=20)
geomean                           7.377k        2.954k       -59.96%

Store filesystem read-only state during mount parsing instead of
serializing and reparsing mount options on every scrape. Rework
textfile metric family conversion to build one desc per family and
normalize label sets by index.

Benchstat:
goos: linux
goarch: amd64
pkg: github.com/prometheus/node_exporter/collector
cpu: AMD Ryzen 9 8945HS w/ Radeon 780M Graphics
                              │ /tmp/base.txt │          /tmp/current.txt           │
                              │    sec/op     │   sec/op     vs base                │
ParseFilesystemLabels-8          219.78µ ± 0%   98.89µ ± 1%  -55.01% (p=0.000 n=20)
TextfileConvertMetricFamily-8     466.0µ ± 1%   223.5µ ± 1%  -52.04% (p=0.000 n=20)
geomean                           320.0µ        148.7µ       -53.55%

                              │ /tmp/base.txt │           /tmp/current.txt           │
                              │     B/op      │     B/op      vs base                │
ParseFilesystemLabels-8         191.29Ki ± 0%   99.28Ki ± 0%  -48.10% (p=0.000 n=20)
TextfileConvertMetricFamily-8    472.0Ki ± 0%   292.8Ki ± 0%  -37.95% (p=0.000 n=20)
geomean                          300.5Ki        170.5Ki       -43.25%

                              │ /tmp/base.txt │          /tmp/current.txt           │
                              │   allocs/op   │  allocs/op   vs base                │
ParseFilesystemLabels-8           4.169k ± 0%   1.097k ± 0%  -73.69% (p=0.000 n=20)
TextfileConvertMetricFamily-8    13.054k ± 0%   7.955k ± 0%  -39.06% (p=0.000 n=20)
geomean                           7.377k        2.954k       -59.96%

Signed-off-by: Kevin Burke <kevin@burke.dev>
@yuvipanda
Copy link
Copy Markdown

We're noticing that on GKE sometimes each scrape takes like, 20-30s, and readonly errors seem to be the primary difference vs our EKS clusters (which are faster). Are you experiencing something like that kinda slowdown?

@SuperQ
Copy link
Copy Markdown
Member

SuperQ commented Apr 25, 2026

Mount options can change between scrapes.

This would cache this value permanently. We need to parse every scrape to find out if someone ran mount -o remount,rw, for example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants