Data Store Metrics Manager repeated queries not updating "ago" time frame

[knoopum]

Describe the bug
It appears that repeated calls to query() for the Data Store Metrics manager are not updating the timeframe based on the ago parameter. For example, when ago is set to 15 (and ago_unit to minutes), and the query is repeated once a minute, the results are always returned for the same time frame on each subsequent call, rather than advancing a minute each time.

The expectation is that "ago" means the query will return the results for the previous period relative to the time at which the query is made. So for ago and ago_unit of 15 and minutes, the result should be a sliding 15-minute window of time. So, if a query is made at 9:00, it should return the results from 8:45-9:00, and if a subsequent call is made at 9:05, it should return the results for 8:50-9:05.

(Noe that there is a several minute lag between the time of a query, and the most recently available data, so a query at 9:00 usually returns data in a range more like 8:45-8:53 or 8:45-8:54.)

The issue is that it appears once a query is made, the timeframe becomes locked, and all subsequent queries -- even minutes later -- still return results for that same initial timeframe, rather than representing a sliding window into the past relative to the time a query is made.

To Reproduce
Steps to reproduce the behavior:

import arcgis
from arcgis.gis import GIS
from datetime import datetime, timezone
import time

# Need to connect with Administrator account for gis.admin to be accessible.
gis = GIS("home")

# Get Data Store Metrics Manager
dsmm = gis.admin.datastore_metrics

# Set the time though which to keep querying the metric, until it is included in what is returned.
stop_at = datetime.now()

# Loop until time range for metrics returned includes the "stop at" time, or exceeded expected number of iterations.
finished = False
iterations = 1
max_iterations = 10
while( not finished and iterations <= max_iterations ):

    # Query a metric for the last 15 minutes.
    metric = dsmm.query(
        metric = arcgis.gis.admin.DataStoreMetric.AVG_CPU,
        bin_size = 1,
        bin_unit = arcgis.gis.admin.DataStoreTimeUnit.MINUTE,
        aggregation = arcgis.gis.admin.DataStoreAggregation.MAX,
        ago = 15,
        ago_unit = arcgis.gis.admin.DataStoreTimeUnit.MINUTE
    )
    
    print(f"Current time: {datetime.now()}, Last record ts: {metric[0]['ts']}, Stop at ts: {stop_at}")
    display(metric)

    # If queried metrics includes the stop_at time, then done, otherwise sleep and try again.
    if( metric[0]['ts'] >= stop_at ):
        finished = True
    else:
        time.sleep(60)
        iterations += 1

print("\nDone")

Output:

Current time: 2026-03-22 21:11:26.589495, Last record ts: 2026-03-22 21:05:00, Stop at ts: 2026-03-22 21:11:25.985634
[{'ts': datetime.datetime(2026, 3, 22, 21, 5), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 4), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 3), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 2), 'value': 3.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 1), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 0), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 59), 'value': 5.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 58), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 57), 'value': 1.0}]
Current time: 2026-03-22 21:12:26.595062, Last record ts: 2026-03-22 21:05:00, Stop at ts: 2026-03-22 21:11:25.985634
[{'ts': datetime.datetime(2026, 3, 22, 21, 5), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 4), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 3), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 2), 'value': 3.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 1), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 0), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 59), 'value': 5.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 58), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 57), 'value': 1.0}]
Current time: 2026-03-22 21:13:26.597782, Last record ts: 2026-03-22 21:05:00, Stop at ts: 2026-03-22 21:11:25.985634
[{'ts': datetime.datetime(2026, 3, 22, 21, 5), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 4), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 3), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 2), 'value': 3.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 1), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 0), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 59), 'value': 5.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 58), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 57), 'value': 1.0}]
Current time: 2026-03-22 21:14:26.599912, Last record ts: 2026-03-22 21:05:00, Stop at ts: 2026-03-22 21:11:25.985634
[{'ts': datetime.datetime(2026, 3, 22, 21, 5), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 4), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 3), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 2), 'value': 3.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 1), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 0), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 59), 'value': 5.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 58), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 57), 'value': 1.0}]
Current time: 2026-03-22 21:15:26.602785, Last record ts: 2026-03-22 21:05:00, Stop at ts: 2026-03-22 21:11:25.985634
[{'ts': datetime.datetime(2026, 3, 22, 21, 5), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 4), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 3), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 2), 'value': 3.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 1), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 0), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 59), 'value': 5.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 58), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 57), 'value': 1.0}]
Current time: 2026-03-22 21:16:26.605988, Last record ts: 2026-03-22 21:05:00, Stop at ts: 2026-03-22 21:11:25.985634
[{'ts': datetime.datetime(2026, 3, 22, 21, 5), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 4), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 3), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 2), 'value': 3.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 1), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 0), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 59), 'value': 5.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 58), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 57), 'value': 1.0}]
Current time: 2026-03-22 21:17:26.608386, Last record ts: 2026-03-22 21:05:00, Stop at ts: 2026-03-22 21:11:25.985634
[{'ts': datetime.datetime(2026, 3, 22, 21, 5), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 4), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 3), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 2), 'value': 3.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 1), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 0), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 59), 'value': 5.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 58), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 57), 'value': 1.0}]
Current time: 2026-03-22 21:18:26.610899, Last record ts: 2026-03-22 21:05:00, Stop at ts: 2026-03-22 21:11:25.985634
[{'ts': datetime.datetime(2026, 3, 22, 21, 5), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 4), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 3), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 2), 'value': 3.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 1), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 0), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 59), 'value': 5.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 58), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 57), 'value': 1.0}]
Current time: 2026-03-22 21:19:26.612860, Last record ts: 2026-03-22 21:05:00, Stop at ts: 2026-03-22 21:11:25.985634
[{'ts': datetime.datetime(2026, 3, 22, 21, 5), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 4), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 3), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 2), 'value': 3.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 1), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 0), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 59), 'value': 5.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 58), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 57), 'value': 1.0}]
Current time: 2026-03-22 21:20:26.614942, Last record ts: 2026-03-22 21:05:00, Stop at ts: 2026-03-22 21:11:25.985634
[{'ts': datetime.datetime(2026, 3, 22, 21, 5), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 4), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 3), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 2), 'value': 3.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 1), 'value': 2.0},
 {'ts': datetime.datetime(2026, 3, 22, 21, 0), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 59), 'value': 5.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 58), 'value': 1.0},
 {'ts': datetime.datetime(2026, 3, 22, 20, 57), 'value': 1.0}]

Done

Note that in the above example, the first query (at 21:12) for the last 15-minutes of data returns data in a time frame of 20:57-21:05, and that the data from each subsequent call one minute later continues to return data from that same timeframe.

Expected behavior
When using the ago and ago_units parameter, I am expecting to be able to define a window of time for the query relative to the time at which the query is run. So if I re-run the query with the same parameters once a minute, I would expect the time window of the results to advance a minute each time as well.

Platform (please complete the following information):

• OS: 'Windows 11
• Browser: Chrome
• Python API Version: 2.4.2 (ArcGIS Online Standard Runtime 13)

[nanaeaubry]

@knoopum Thanks for reporting this, we will dig into it and see what we can find

[kgalliher]

@knoopum I am able to show what you see with your code but I have some questions.

1. Can you verify that there is activity in you org while the script is running?
2. Can you run the script again but include the start_time parameter using datetime.now()?

I noticed that the web interface is also slow to react to new activity. However, with the start_time set, I can see the changes occur in the script output. That output is corresponding to the graph in the web interface.

[knoopum]

@kgalliher

1. Yes, I can verify that there is activity in the org while running this test script. I can also see from re-opening the Resource usage chart in the user interface every minute or so, that it is updating with new data every few minutes during the test.
2. When using the start_time and end_time parameters (or just start_time), the method behaves as expected.

[knoopum]

@kgalliher

Unfortunately, it looks like there are certain scenarios in which the start_time and end_time parameters fail in the same way, not changing what they return, even after new data becomes available for the time range of the query.

If you query once a minute for the same recent start_time and end_time range, waiting for data to become available for the whole time range, then the results never change. It never picks up new data as it becomes available.

If you update the query to add a microsecond to the end_time, then the results do pick-up new data as it becomes available for the whole time range.

It seems like the issue is that if the start_time and end_time are the same as a previous call, then a new query isn't actually made, and it returns the cached results of the last query? However, if you change the start_time and/or the end_time, then it does make a new query.

Below is some sample code that illustrates the issue. It attempts to gather the data store metrics for the next five minute time range from when it is started. When the start_time and end_time are the same on each pass, then the results never change; however, if you uncomment the line that adds a microsecond to the end_time on each pass, then the results do update as new data becomes available in the time range.

import arcgis
from arcgis.gis import GIS
from datetime import datetime, timedelta, timezone
import pandas

# Need to connect with Administrator account for gis.admin to be accessible.
gis = GIS("home")

# Get Data Store Metrics Manager
dsmm = gis.admin.datastore_metrics

# Set start_time to now
start_time = datetime.now(tz=timezone.utc)
print(f"start_time: {start_time}")

# Set end_time into the future
end_time = start_time + timedelta(minutes=5)
print(f"end_time:   {end_time}")

# Loop until time range for metrics returned includes the stop_time, or exceeded expected number of iterations.
finished = False
iteration = 1
max_iterations = 20
while( not finished and iteration <= max_iterations ):

    print(f"\nIteration: {iteration} ({datetime.now(tz=timezone.utc)})")

    # Query a metric up to the buffer_end_time.
    metric = dsmm.query(
        metric = arcgis.gis.admin.DataStoreMetric.AVG_CPU,
        bin_size = 1,
        bin_unit = arcgis.gis.admin.DataStoreTimeUnit.MINUTE,
        aggregation = arcgis.gis.admin.DataStoreAggregation.MAX,
        start_time = start_time,
        end_time = end_time
    )

    # Any metrics returned?
    if(metric):
        display(pandas.DataFrame(metric))
    
        # If querired metrics includes the end_time, then done, otherwise sleep and try again.
        first_ts = metric[-1]['ts'].replace(tzinfo=timezone.utc)
        timezone_offset = int((start_time - first_ts).seconds / 3600)
        if( metric[0]['ts'].replace(tzinfo=timezone.utc) + timedelta(hours=timezone_offset) >= end_time ):
            finished = True
    else:
        print("No metrics available for query range yet.")

    # If not done, increment iteration counter and pause
    if( not finished ):
        iteration += 1
        time.sleep(60)

        # When commented out, the query results never change for the same time range.
        # When uncommented, the end_time is updated for each query, and the query results do change as
        # as new data becomes available for the time range.
        # end_time += timedelta(microseconds=1)  

print("\nDone")