HDDS-14446. Clarify Datanode disk space related commands and metrics #9646

siddhantsangwan · 2026-01-19T07:59:36Z

What changes were proposed in this pull request?

Please see the jira for the problem description.

Changes proposed:

In Datanode volume storage reports, also include raw (the actual capacity and available space for a volume) file system stats: Filesystem Capacity, Filesystem Available. These are different from the capacity and available space that's currently reported:

capacity = fsCapacity - du.reserved

This is what the new proto message looks like:

message StorageReportProto {
  required string storageUuid = 1;
  required string storageLocation = 2;
  optional uint64 capacity = 3 [default = 0];
  optional uint64 scmUsed = 4 [default = 0];
  optional uint64 remaining = 5 [default = 0];
  optional StorageTypeProto storageType = 6 [default = DISK];
  optional bool failed = 7 [default = false];
  optional uint64 committed = 8 [default = 0];
  optional uint64 freeSpaceToSpare = 9 [default = 0];
  optional uint64 reserved = 10;
  /*
   Raw filesystem stats (as reported by the local filesystem). These represent the real device
   capacity/available, independent of Ozone's reserved-space adjustment.
   */
  optional uint64 fsCapacity = 11 [default = 0];
  optional uint64 fsAvailable = 12 [default = 0];
}

Note fsUsed is not reported in the heartbeat as it can be calculated where needed using fsCapacity - fsAvailable.

These stats are then updated at various places: Datanode JMX metrics, SCM JMX metrics, usageinfo command, and Recon Datanodes page mouse hover screen.
Instead of using the ambiguous term capacity, now we clearly differentiate by using Ozone Capacity to mean Filesystem Capacity - du.reserved etc.

What's remaining (to be done in subsequent pull requests):

Recon main overview page.
SCM Web UI main page.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14446

How was this patch tested?

Edited and added various unit tests.

The Recon overview page is still the same as before:

`usageinfo` command shows the new names and the new fs stats:

bash-5.1$ ozone admin datanode usageinfo -c=5 -m
Usage Information (3 Datanodes)

UUID                    : 5953e64b-1acc-4cf3-9aaf-aedb271a02fb 
IP Address              : 172.18.0.9 
Hostname                : ozone-datanode-2.ozone_default 
Filesystem Capacity     : 1081100128256 B (1006.85 GB) 
Filesystem Used         : 68737728512 B (64.02 GB) 
Filesystem Used %       : 6.36% (Filesystem Used/Filesystem Capacity) 
Filesystem Available    : 1012362399744 B (942.84 GB) 
Filesystem Available %  : 93.64% (Filesystem Available/Filesystem Capacity) 
Ozone Capacity          : 1080992018248 B (1006.75 GB) 
Ozone Used              : 5812224 B (5.54 MB) 
Ozone Used %            : 0.00% (Ozone Used/Ozone Capacity) 
Ozone Available         : 1012362399744 B (942.84 GB) 
Ozone Available %       : 93.65% (Ozone Available/Ozone capacity) 
Pipeline(s)             : 2 
Container(s)            : 0 
Container Pre-allocated : 0 B (0 B) 
Remaining Allocatable   : 1012362399744 B (942.84 GB) 
Free Space To Spare     : 104857600 B (100 MB) 
Reserved                : 108110008 B (103.10 MB) 

UUID                    : 75b9772e-61bc-4b79-b93c-807584ee604c 
IP Address              : 172.18.0.8 
Hostname                : ozone-datanode-1.ozone_default 
Filesystem Capacity     : 1081100128256 B (1006.85 GB) 
Filesystem Used         : 68737728512 B (64.02 GB) 
Filesystem Used %       : 6.36% (Filesystem Used/Filesystem Capacity) 
Filesystem Available    : 1012362399744 B (942.84 GB) 
Filesystem Available %  : 93.64% (Filesystem Available/Filesystem Capacity) 
Ozone Capacity          : 1080992018248 B (1006.75 GB) 
Ozone Used              : 5812224 B (5.54 MB) 
Ozone Used %            : 0.00% (Ozone Used/Ozone Capacity) 
Ozone Available         : 1012362399744 B (942.84 GB) 
Ozone Available %       : 93.65% (Ozone Available/Ozone capacity) 
Pipeline(s)             : 2 
Container(s)            : 0 
Container Pre-allocated : 0 B (0 B) 
Remaining Allocatable   : 1012362399744 B (942.84 GB) 
Free Space To Spare     : 104857600 B (100 MB) 
Reserved                : 108110008 B (103.10 MB) 

UUID                    : e9587d3d-200e-4186-b983-9fde5a4c9a40 
IP Address              : 172.18.0.2 
Hostname                : ozone-datanode-3.ozone_default 
Filesystem Capacity     : 1081100128256 B (1006.85 GB) 
Filesystem Used         : 68737728512 B (64.02 GB) 
Filesystem Used %       : 6.36% (Filesystem Used/Filesystem Capacity) 
Filesystem Available    : 1012362399744 B (942.84 GB) 
Filesystem Available %  : 93.64% (Filesystem Available/Filesystem Capacity) 
Ozone Capacity          : 1080992018248 B (1006.75 GB) 
Ozone Used              : 5812224 B (5.54 MB) 
Ozone Used %            : 0.00% (Ozone Used/Ozone Capacity) 
Ozone Available         : 1012362399744 B (942.84 GB) 
Ozone Available %       : 93.65% (Ozone Available/Ozone capacity) 
Pipeline(s)             : 2 
Container(s)            : 0 
Container Pre-allocated : 0 B (0 B) 
Remaining Allocatable   : 1012362399744 B (942.84 GB) 
Free Space To Spare     : 104857600 B (100 MB) 
Reserved                : 108110008 B (103.10 MB)

SCM JMX metrics:

{
    "name" : "Hadoop:service=StorageContainerManager,name=SCMNodeMetrics",
    "modelerType" : "SCMNodeMetrics",
    "tag.Context" : "ozone",
    "tag.Hostname" : "d5f1173fd92a",
    "InMaintenanceStaleNodes" : 0,
    "InMaintenanceHealthyReadonlyNodes" : 0,
    "InMaintenanceHealthyNodes" : 0,
    "InMaintenanceDeadNodes" : 0,
    "InServiceStaleNodes" : 0,
    "InServiceHealthyReadonlyNodes" : 0,
    "InServiceHealthyNodes" : 3,
    "InServiceDeadNodes" : 0,
    "DecommissioningStaleNodes" : 0,
    "DecommissioningHealthyReadonlyNodes" : 0,
    "DecommissioningHealthyNodes" : 0,
    "DecommissioningDeadNodes" : 0,
    "DecommissionedStaleNodes" : 0,
    "DecommissionedHealthyReadonlyNodes" : 0,
    "DecommissionedHealthyNodes" : 0,
    "DecommissionedDeadNodes" : 0,
    "EnteringMaintenanceStaleNodes" : 0,
    "EnteringMaintenanceHealthyReadonlyNodes" : 0,
    "EnteringMaintenanceHealthyNodes" : 0,
    "EnteringMaintenanceDeadNodes" : 0,
    "AllNodes" : 3,
    "NonWritableNodes" : 0,
    "TotalOzoneCapacity" : 3242976054744,
    "DecommissionedDiskRemaining" : 0,
    "DecommissionedSSDRemaining" : 0,
    "MaintenanceSSDRemaining" : 0,
    "SSDCapacity" : 0,
    "DecommissionedDiskCapacity" : 0,
    "SSDUsed" : 0,
    "MaintenanceDiskCapacity" : 0,
    "TotalFilesystemCapacity" : 3243300384768,
    "TotalOzoneUsed" : 17436672,
    "DiskCapacity" : 3242976054744,
    "DecommissionedDiskUsed" : 0,
    "TotalFilesystemUsed" : 206214365184,
    "TotalFilesystemAvailable" : 3037086019584,
    "DecommissionedSSDUsed" : 0,
    "MaintenanceDiskUsed" : 0,
    "SSDRemaining" : 0,
    "DecommissionedSSDCapacity" : 0,
    "DiskRemaining" : 3037086019584,
    "DiskUsed" : 17436672,
    "MaintenanceDiskRemaining" : 0,
    "MaintenanceSSDCapacity" : 0,
    "MaintenanceSSDUsed" : 0,
    "NumHBProcessed" : 33192,
    "NumHBProcessingFailed" : 0,
    "NumNodeCommandQueueReportProcessed" : 33192,
    "NumNodeCommandQueueReportProcessingFailed" : 0,
    "NumNodeReportProcessed" : 2763,
    "NumNodeReportProcessingFailed" : 0
  }

Datanode Web UI main page (ozone-datanode-1):

Note that Total Capacity shown above should be the same as Filesystem Capacity. I've retained Total Capacity just because it was already there. We could remove it if needed.

Datanode JMX metrics (ozone-datanode-1):

{
    "name" : "Hadoop:service=HddsDatanode,name=VolumeInfoMetrics-/data/hdds",
    "modelerType" : "VolumeInfoMetrics-/data/hdds",
    "tag.Context" : "ozone",
    "tag.StorageType" : "DISK",
    "tag.DatanodeUuid" : "75b9772e-61bc-4b79-b93c-807584ee604c",
    "tag.VolumeType" : "DATA_VOLUME",
    "tag.StorageDirectory" : "/data/hdds/hdds",
    "tag.VolumeState" : "NORMAL",
    "tag.Hostname" : "9cb7bc189a70",
    "AvailableSpaceInsufficient" : 0,
    "DbCompactLatencyNumOps" : 7,
    "DbCompactLatencyAvgTime" : 0.0,
    "NumScans" : 16,
    "NumScansSkipped" : 0,
    "ReservedCrossesLimit" : 1,
    "Committed" : 0,
    "Containers" : 0,
    "LayoutVersion" : 1,
    "OzoneCapacity" : 1080992018248,
    "OzoneAvailable" : 1012361478144,
    "OzoneUsed" : 5812224,
    "Reserved" : 108110008,
    "TotalCapacity" : 1081100128256,
    "FilesystemCapacity" : 1081100128256,
    "FilesystemAvailable" : 1012361478144,
    "FilesystemUsed" : 68738650112
  }

Recon Datanode page

PR is draft while CI runs in my fork.

adoroszlai · 2026-01-19T12:48:42Z

...ainer-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeUsage.java

+  /**
+   * @return raw filesystem capacity (cached) for the configured volume path.
+   */
+  public long getFsCapacity() {
+    return source.getCapacity();
+  }
+
+  /**
+   * @return raw filesystem available space (cached) for the configured volume path.
+   */
+  public long getFsAvailable() {
+    return source.getAvailable();
+  }
+


Thanks @siddhantsangwan for the patch.

Instead of adding these new methods in VolumeUsage, I suggest getting the same values via realUsage() in a single call, and passing that to getCurrentUsage().

HDDS-14446.patch

Thanks for the patch, I updated the PR.

adoroszlai · 2026-01-20T14:02:21Z

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java

+  /**
+   * Simple class for grouping disk space related values.
+   */
+  public static final class FsUsageTotals {
+    private final long capacity;
+    private final long available;
+    private final long used;
+
+    private FsUsageTotals(long capacity, long available, long used) {
+      this.capacity = capacity;
+      this.available = available;
+      this.used = used;
+    }
+
+    public long getCapacity() {
+      return capacity;
+    }
+
+    public long getAvailable() {
+      return available;
+    }
+
+    public long getUsed() {
+      return used;
+    }
+  }
+


Sorry, I missed this earlier: can we use SpaceUsageSource.Fixed instead?

ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/SpaceUsageSource.java

Lines 48 to 77 in 65f0af0

/**

* A static source of space usage. Can be a point in time snapshot of a

* real volume usage, or can be used for testing.

*/

final class Fixed implements SpaceUsageSource {

private final long capacity;

private final long available;

private final long used;

public Fixed(long capacity, long available, long used) {

this.capacity = capacity;

this.available = Math.max(Math.min(available, capacity - used), 0);

this.used = used;

}

@Override

public long getCapacity() {

return capacity;

}

@Override

public long getAvailable() {

return available;

}

@Override

public long getUsedSpace() {

return used;

}

Makes sense - updated.

Thanks. Can you please check acceptance test failure? Happened for both:

siddhantsangwan@aba71ea

siddhantsangwan@773f405

Yes the failures were related. Since the json output for usageinfo changed, the related robot test was failing. I pushed a fix to datanode.robot. CI is passing now - https://github.com/siddhantsangwan/ozone/actions/runs/21211360808.

adoroszlai

Thanks @siddhantsangwan for updating the patch, LGTM. Let's wait for others to take a look, too.

sodonnel

LGTM, although I am not too familiar with this area.

priyeshkaratha

Thanks @siddhantsangwan for working on this. Overall code LGTM.

priyeshkaratha · 2026-01-22T16:31:26Z

...service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoMetrics.java


  private static final MetricsInfo CAPACITY =
-      Interns.info("Capacity", "Capacity");
+      Interns.info("OzoneCapacity", "Ozone usable capacity (after reserved space adjustment)");


Even though the new naming conventions make more sense, this is a breaking change for existing metrics consumers. They will need to update their usage (for example, from Capacity to OzoneCapacity).

How are such breaking changes tracked and communicated? Is there any documentation or process to notify consumers that they need to make updates? For example, if system tests depend on these metrics, how do we ensure these changes are identified and handled?

Generally, we have annotations in the code to mark the stability and intended audience of an interface. See InterfaceAudience.java and InterfaceStability.java. For example, SCMNodeMetrics is marked as @InterfaceAudience.Private which means it's Unstable as per the documentation in:

Classes that are {@link InterfaceAudience.Private} are to be considered unstable

/** * No guarantee is provided as to reliability or stability across any * level of release granularity. */ @Documented @Retention(RetentionPolicy.RUNTIME) public @interface Unstable { }

I considered this when making this change. The options were:

To not change the existing names at all and only introduce new terms Filesystem Capacity etc. This wouldn't have solved the confusion.

To keep both - the existing terms capacity, available and scmUsed and also introduce ozoneCapacity, ozoneAvailable, ozoneUsed. This would have caused more confusion.

Change the existing names and introduce fs stats, which is what I did in this PR.

Moreover, I think this change is required for correctness - see https://issues.apache.org/jira/browse/HDDS-14388. Currently even ozone devs find it hard to understand these stats and have to spend a non trivial amount of time to figure them out. Overall, I think we should try to make them clear across the ozone code.

devabhishekpal

Thanks @siddhantsangwan for this improvement.
I just had one nit, but overall this patch LGTM, +1

devabhishekpal · 2026-01-27T06:56:30Z

...src/main/resources/webapps/recon/ozone-recon-web/src/v2/types/datanodeStorageReport.types.ts

@@ -0,0 +1,36 @@
+/*


Just a nit, can we move this to overview.types or datanode.types instead of an individual file?

Can this be taken up in https://issues.apache.org/jira/browse/HDDS-14483?

@devabhishekpal let's handle this in HDDS-14483 since that involves Recon overview pages changes.

siddhantsangwan · 2026-01-27T10:28:24Z

Thanks for the reviews everyone. Merging this now that CI is green.

siddhantsangwan added 2 commits January 19, 2026 12:50

HDDS-14446. Make Datanode disk space related commands and metrics clear

d408eed

lower log level and improve aggregation logic

c4abe79

siddhantsangwan requested review from devabhishekpal, sodonnel and sumitagrawl January 19, 2026 08:56

adoroszlai reviewed Jan 19, 2026

View reviewed changes

adoroszlai changed the title ~~HDDS-14446. Make Datanode disk space related commands and metrics clear~~ HDDS-14446. Clarify Datanode disk space related commands and metrics Jan 20, 2026

siddhantsangwan added 2 commits January 20, 2026 18:27

use realUsage()

379c0ab

fix checkstyle

aba71ea

adoroszlai reviewed Jan 20, 2026

View reviewed changes

siddhantsangwan added 3 commits January 21, 2026 12:44

use SpaceUsageSource.Fixed instead

86eab9f

add unit test for SCMNodeManager#getTotalFilesystemUsage

773f405

change datanode.robot

4ee9782

adoroszlai reviewed Jan 22, 2026

View reviewed changes

sodonnel approved these changes Jan 22, 2026

View reviewed changes

priyeshkaratha reviewed Jan 22, 2026

View reviewed changes

devabhishekpal approved these changes Jan 27, 2026

View reviewed changes

siddhantsangwan marked this pull request as ready for review January 27, 2026 08:11

siddhantsangwan merged commit 1f4b3e0 into apache:master Jan 27, 2026
57 checks passed

	/**
	* A static source of space usage. Can be a point in time snapshot of a
	* real volume usage, or can be used for testing.
	*/
	final class Fixed implements SpaceUsageSource {

	private final long capacity;
	private final long available;
	private final long used;

	public Fixed(long capacity, long available, long used) {
	this.capacity = capacity;
	this.available = Math.max(Math.min(available, capacity - used), 0);
	this.used = used;
	}

	@Override
	public long getCapacity() {
	return capacity;
	}

	@Override
	public long getAvailable() {
	return available;
	}

	@Override
	public long getUsedSpace() {
	return used;
	}

HDDS-14446. Clarify Datanode disk space related commands and metrics #9646

HDDS-14446. Clarify Datanode disk space related commands and metrics #9646

Uh oh!

Conversation

siddhantsangwan commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

The Recon overview page is still the same as before:

usageinfo command shows the new names and the new fs stats:

SCM JMX metrics:

Datanode Web UI main page (ozone-datanode-1):

Datanode JMX metrics (ozone-datanode-1):

Recon Datanode page

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

sodonnel left a comment

Choose a reason for hiding this comment

Uh oh!

priyeshkaratha left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devabhishekpal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

siddhantsangwan commented Jan 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

siddhantsangwan commented Jan 19, 2026 •

edited

Loading

`usageinfo` command shows the new names and the new fs stats: