Defer redis connection until first use (avoid sockets stuck in CLOSE_WAIT)#3618
Defer redis connection until first use (avoid sockets stuck in CLOSE_WAIT)#3618jes wants to merge 1 commit intoOpenSIPS:masterfrom
Conversation
liviuchircu
left a comment
There was a problem hiding this comment.
Hey @jes! I would be onboard with this with no hesitation, were it not impacting the startup phase. As of now, say you start OpenSIPS with Redis offline -- it gives an Error and fails startup. If we merge this, there is an added side-effect of OpenSIPS now successfully starting with an offline Redis.
Is this subtle startup behavior change something we want? Is it for the better? And, also worth asking, are the current, "indefinitely" temporary CLOSE_WAIT sockets that harmful?
|
I've been running a fork that solves the same CLOSE_WAIT problem but addresses @liviuchircu's concern about the startup behavior change. Instead of unconditionally deferring the connection, add an opt-in module parameter: /* in cachedb_redis.c, alongside shutdown_on_error */ /* in the param_export_t array */ Then the change in redis_new_connection() becomes: /* if doing failover Redises, only connect the 1st one for now! */ This way:
|
Summary
We were seeing OpenSIPS leaving redis client sockets in the CLOSE_WAIT state when redis restarts. This PR prevents this problem.
Details
When redis restarts, the client sockets go into the CLOSE_WAIT state until OpenSIPS next tries to use the socket and then notices that it needs to reconnect.
Some of the OpenSIPS processes open redis sockets that they never use, because the connection is opened immediately even if it doesn't get used.
This is not a major issue, but it means we keep getting alerts for sockets stuck in CLOSE_WAIT.
Solution
This PR fixes the problem by deferring the connection to redis until it is actually used. That way the only sockets that go into CLOSE_WAIT are ones that have been actively used at least once, so we can expect them to get used again, which will clean up the CLOSE_WAIT sockets.
Compatibility
I don't imagine this breaks any scenarios.
Closing issues
N/A