Add inclusion and exclusion filter for log types in the Syslog Agent

[chombium]

There are some use cases where some app owners might not need all types of logs: RTR, APP, STG etc. It would be nice if inclusion and exclusion filters could be added in order that the downstream consumers are not overloaded with unneeded data. This would also save some costs for outgoing network traffic for the operators.

As there already is a drain-data Syslog Drain URL parameter, the extension with exclude-log-types parameter is easy to implement. Theexclude-log-types parameter should only be taken into account when there is no data filtering by type, no drain-data or drain-type parameter is set or when the value of these parameters is logs. The values for the exclude-log-types should be lower case variants of the log types listed in the link above: app,rtr,stg,cell, etc. Multiple values for the log types should be possible to set: exclude-log-types=rtr,cell,stg

The validation and parsing of the data type is done in the getBindingType and the actual filtering sending or skipping log envelopes in the Write and sendsLogs functions. This should be extended to take the log type exclusion filter into account.

The inspection of log type for the log envelopes can be done with the source_type tag.

To check the logs and the source types, one can use the following command of the log-cache cli plugin:

 cf tail -n 10 -c logs --json <app_guid>|jq '.batch[].tags.source_type
"APP/PROC/WEB"
"APP/PROC/WEB"
"APP/PROC/WEB"
"APP/PROC/WEB"
"RTR"
"APP/PROC/WEB"
"APP/PROC/WEB"
"RTR"

The filtering of application logs might be a bit costly as it will involve string parsing. We should take the performance into account and decide if:

• we should put the app logs filtering behind a feature flag, so that the cf operators explicitly are aware about the possible performance degradation and want to to have that
• say that by definition the application logs cannot be filtered out, as the app developers would be most interested in them

Example setup usage:
exclude RTR and CELL logs:

cf cups -l https://my-syslog-server.com?exclude-log-types=rtr,cell

[bennygoerzig]

Should we implement an inclusion filter where you specify which log types to keep (e.g., include-log-types=app,rtr) or an exclusion filter where you specify which types to filter out (e.g., exclude-log-types=stg,cell)? Both would default to sending all logs when not specified, and the filtering would only apply when drain-type is not set or equals logs. I'm curious which approach feels more intuitive for CF app developers and which would be easier for service providers to configure when constructing syslog drain URLs on the broker side.

[chombium]

@bennygoerzig that's what I was thinking as well, should we have an include or exclude filter. I've proposed an exclusion filter as IMO the app developers should be aware about everything that is happening to their apps and they would either want all logs or some sub-set with some log types filtered out. Of course, the end effect is the same despite how we implement the filter.

The question is which way would be better:

1. if someone want's only app logs, should they write

cf cups -l https://my-syslog-server.com?include-log-types=app

or

cf cups -l https://my-syslog-server.com?exclude-log-types=rtr,cell,stg...

2. If someone wants everything but router logs:

cf cups -l https://my-syslog-server.com?include-log-types=app,stg,cell...

or

cf cups -l https://my-syslog-server.com?exclude-log-types=rtr

Thinking out loudly: the CF API has include filter.

Do we want to see the sent data from observability point of view or we want to be compatible with the CF API?

[jorbaum]

@bennygoerzig good question regarding exclusion vs. inclusion and good sketch from @chombium

AFAICS there are common use cases for both only including certain log types and only excluding certain log types. Also the CF API docs state it supports exclude filter for "some fields" e.g. GET /v3/audit_events?target_guids[not]=guid-1,guid-2. However, I personally do not like using [not] as it requires URL encoding (i.e. log-types%5Bnot%5D=)

My take on this would be that we should try to be consistent with the design of the cf create-user-provided-service command. Here this will be the first URL Query param to support a list of values. Both drain-type and drain-data are set to a single value only like all or metrics. I think for the log type it makes sense to give users more flexibility though and therefore supplying a comma-separated list should be a good idea.

How about adding support for both ways to configure and aligning this some more with CF API v3 usage (taking previous example from @chombium ) :

• cf cups -l https://my-syslog-server.com?include-log-types=app or cf cups -l https://my-syslog-server.com?include-log-types=app,stg,cell...
• cf cups -l https://my-syslog-server.com?exclude-log-types=rtr,cell,stg... or cf cups -l https://my-syslog-server.com?exclude-log-types=rtr

This would bring in some additional complexity as users may specify both at the same time. We then should add validation making sure one of include-log-types or exclude-log-types got supplied.

Update: Calling it include-log-types and exclude-log-types instead of log-types and log-types-not as it is more common naming.

[chombium]

I had a discussion with @jorbaum on how to implement the filter and we have decided to make both include-log-types and exclude-log-types query parameters available as they both have their pros and cons. Only one parameter can be used in the Syslog Drain URL. If both parameters are included there will be an error thrown and the drain will be set as invalid.
Combining this change with #633 the app developer will be informed immediately about the error they've made.

[KarstenSchnitter]

The discussion has mentioned the APP source type just by this three letter acronym. In reality, this is just the prefix for a bunch of possible source types. The most frequent one is APP/PROC/WEB as this is the most common process. But there might be different ones for jobs, sidecars, or other processes belonging to the application. What is the idea on dealing with those logs?

I would be fine with categorizing them as APP source type. But maybe a better approach would be to support prefixes on the include and exclude filters to give more control on those subtypes. This could be done explicitly with support of a wildcard "*" ( maybe only at the end) or implicitly by taking the user-provided string and just do a "starts-with" comparison.

[chombium]

Hi @KarstenSchnitter,

The most frequent one is APP/PROC/WEB as this is the most common process. But there might be different ones for jobs, sidecars, or other processes belonging to the application. What is the idea on dealing with those logs?

That's a very good point. We've discussed about it and decided for the first implementation that everything will be categorized as an app log. The app devs should at least be interested in the logs of all components running in the same container. If the app devs complain about having too many logs from sidecars, jobs and other processes, they should be aware about that and re-configure the loggers of that components.

Thinking about the technical implementation, having substring/regex matching in any form can introduce performance degradation. We can test and see if we could find a solution with good enough performance.

On the other hand, if we add substring/regex matching on log types or even log message text, would mean that we will mask the problem of their solution (app, sidecar, process etc.) logging to much unnecessary data.