Bug: Calculation of inflight events does not consider dropped documents

[saiiman]

The calculation of inflight events is incorrect if documents are intentionally discarded in the logstash pipelines via the "drop" processor.

The method "calculateInflightEvents" (

check_logstash/cmd/pipeline.go

Line 122 in 4bb9291

inflightEvents := calculateInflightEvents(pipe.Events.In, pipe.Events.Out)

) determines the current inflight events based on the global IN / OUT values. If documents are dropped in the pipeline, the values gradually diverge.

Steps to reproduce

1. create a pipeline with a “drop” processor
2. analyze the pipeline statistics via “localhost:9600/_node/stats/pipeline” logstash api.
3. compare the values with the output of the Icinga check

[martialblog]

Hi, from what I can tell, the "inflight events" were a custom metric that the original check plugin used, based on this calculation:

check_logstash/legacy/check_logstash

Line 363 in 4bb9291

inflight_events = events_in - events_out

When writing the Golang code, I decided to reuse this logic to keep compatibility with the old code. Generally I would say, it always tried to be an approximation, not sure though (I didn't write the original plugin), @widhalmt might have more information.

I'd say, where possible use the Pipeline Flow Metrics for newer Logstash versions.

[widhalmt]

@martialblog you're right: It always was an approximation. Full disclosure: I just didn't think of the possibility of dropping events when writing the initial version.

I'd really like to see an improvement like a more accurate representation. I'm just not able to write Golang code, yet. And, honestly, I think I'm not the right person to implement this.

While I can see a potential benefit of seeing the difference of incoming and outgoing events, it's definitely wrong to keep them as inflight_events.

[martialblog]

We currently use the Node Stats API to get the in/out https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#pipeline-stats

Not sure if we can get the information we require from this API. There's was a discussion about this already: #44

If someone has an idea where to get the required info from the API it should be possible to update the calculation.

[martialblog]

I update the docs to make this "inflight events" metric clearer.

#125

I'd say: Pipeline Flow Metrics is the way to go. I'm closing this for now