node_filesystem_readonly reports writable filesystem as read-only after upgrade to 1.12.0/1.12.1

[Merlijnv]

node_filesystem_readonly reports writable filesystem as read-only after upgrade to 1.12.0/1.12.1

Bug report

After upgrading node_exporter from 1.11.x to 1.12.0 or 1.12.1, I am seeing multiple false positives for node_filesystem_readonly.

The most concerning example is /var/data, which is clearly mounted read-write on the host and accepts write operations, yet node_exporter reports it as read-only.

Environment

Docker Compose

node-exporter:
  image: prom/node-exporter:${NODE_EXPORTER_VERSION}
  user: "${NODE_EXPORTER_UID}:${NODE_EXPORTER_GID}"
  container_name: node-exporter
  restart: unless-stopped
  network_mode: host
  command:
    - --path.rootfs=/host
    - --web.listen-address=:9100
    - --collector.textfile.directory=/textfile_collector
  volumes:
    - /:/host:ro,rslave
    - ${NODE_EXPORTER_TEXTFILE_DIR:?NODE_EXPORTER_TEXTFILE_DIR must be set in .env}:/textfile_collector:ro

Observed metrics

Among others, I am seeing the following metric:

node_filesystem_readonly{
  device="/dev/mapper/data--vg0-data--lv--0",
  fstype="ext4",
  mountpoint="/var/data"
} 1

There are also several entries with device_error="permission denied" and device_error="no such file or directory".

Additional verification

The filesystem is mounted as read-write on the host.

findmnt

findmnt -no SOURCE,TARGET,FSTYPE,OPTIONS /var/data

Output:

/dev/mapper/data--vg0-data--lv--0 /var/data ext4 rw,relatime

mount

mount | grep ' /var/data '

Output:

/dev/mapper/data--vg0-data--lv--0 on /var/data type ext4 (rw,relatime)

The mount options clearly show rw.

Write test

I can successfully write to the filesystem:

touch /var/data/node_exporter_rw_test
echo test > /var/data/node_exporter_rw_test
cat /var/data/node_exporter_rw_test
rm -f /var/data/node_exporter_rw_test

Output:

test

All operations complete successfully.

Expected behavior

node_filesystem_readonly{mountpoint="/var/data"} 0

Actual behavior

node_filesystem_readonly{mountpoint="/var/data"} 1

Version comparison

Version	Result
1.11.x	Works as expected
1.12.0	Reports /var/data as read-only
1.12.1	Reports /var/data as read-only

Related issue

I found issue #2506:

#2506

However, that issue appears to be related to the host root filesystem being bind-mounted read-only into the container, causing node_exporter to observe the filesystem as read-only from inside the container. 1

My situation appears different:

• /var/data is mounted as rw on the host.
• Write operations succeed.
• The issue only appears after upgrading from 1.11.x to 1.12.x.
• Multiple filesystems are now being reported as read-only or returning device errors.

Has anything changed in the filesystem collector between 1.11.x and 1.12.x that could affect how read-only status is determined for bind-mounted root filesystems in containers?

I am happy to provide additional debugging information if needed.