what is the reasonable setting for expiry-interval and flush-interval?

[barrowkwan]

Hi,
I am not sure if this is gostatsd issue ( still investigate ), we have missing metrics and wonder if it is because of our expiry-interval (1h) and flush-interval (10s) setting. Could this cause gostatsd remove metric in some cases ? might be it running out of memory ?

thanks

[tiedotguy]

Hi

It's possible, if the metrics are badly tagged (ie, with ephemeral data like a request id or a timestamp), then those will build up in memory over time. It won't drop a metric (unless it's due for expiry), but it could run out of memory and crash. If your system automatically restarts the process, you may not realize it occurred. There's an event emitted on startup which you can look for (there's also one on shutdown, however running out of memory will cause it to not be emitted). Also most of the metrics emitted by gostatsd are accumulated gauges, so you'll see them reset to 0.

Generally speaking, my preference for expiry-internal is -1 (expire immediately). I've found that persistent metrics tend to be confusing for users, with the primary one being "my service has terminated, but there's still data".

By not persisting metrics, it puts the onus on the user to emit data if they want it to persist, conversely when you do persist, then users can't opt-out of it.

It can also be confusing with timers - what value should be emitted for the upper value, when the count is 0? (#135 is related)

Hope this is useful.

[MovieStoreGuy]

Mmm, this is interesting.

I haven't had the chance to look, but is this the default value for gostatsd?
Should we add some further clarifying docs on it to help those in future that would want to explore this?