EMFILE errors

[steina1989]

Environment

• Telemetry Streaming Version: 1.37.0
• BIG-IP Version: 17.1.1.4

Summary

I have a simple Telemetry Streaming setup with Request Logging profiles that delivers the logs to HTTP consumers (logstash).

After a few minutes of delivering the request logs to the consumer (300-500req/s), restnoded.log gets flooded with EMFILE errors and the f5-rest-node process then gets stuck at high CPU and stops delivering any request logs.

A "bigstart restart restnoded restjavad" resolves the issue only for a few minutes.

The impact is loss of request logs, and uncertainty about how this condition affects more important aspects of the box like health monitoring, etc.

Steps To Reproduce

Steps to reproduce the behavior:

1. Submit the following declaration:

{
    "class": "Telemetry",
    "controls": {
        "class": "Controls",
        "memoryThresholdPercent": 90
    },
    "prd": {
        "class": "Telemetry_Namespace",
        "My_Listener": {
            "class": "Telemetry_Listener",
            "port": 6511
        },
        "My_ELK_Consumer": {
            "class": "Telemetry_Consumer",
            "type": "Generic_HTTP",
            "host": "f5-logs-prd.k8s.inf.com",
            "protocol": "https",
            "port": 443,
            "path": "/",
            "headers": [
                {
                    "name": "content-type",
                    "value": "application/json"
                }
            ],
            "outputMode": "processed",
            "allowSelfSignedCert": true
        }
    }
...

2. Create the local listener virtual server, pool, and have a request logging profile reference the pool. Generate traffic.
3. Observe the following error messages in restnode.log:

Wed, 22 Jan 2025 10:46:27 GMT - severe: [telemetry.service.ConsumersService.Generic_HTTP.prd::My_ELK_Consumer] Unexpected error: Error: HTTP Error: connect EMFILE 172.19.16.239:443 - Local (undefined:undefined)
Message: HTTP Error: connect EMFILE 172.19.16.239:443 - Local (undefined:undefined)
Traceback:
Error: connect EMFILE 172.19.16.239:443 - Local (undefined:undefined)
    at Object._errnoException (util.js:1022:11)
    at _exceptionWithHostPort (util.js:1044:20)
    at internalConnect (net.js:976:16)
    at defaultTriggerAsyncIdScope (internal/async_hooks.js:283:19)
    at GetAddrInfoReqWrap.emitLookup [as callback] (net.js:1122:9)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:97:10)