Skip to content

RDKEMW-15927 : [Xione DE] rbus self-deadlock in XConf privacy mode fetch causing ~188s T2 init delay#303

Open
tabbas651 wants to merge 10 commits intodevelopfrom
topic/RDKEMW-15007
Open

RDKEMW-15927 : [Xione DE] rbus self-deadlock in XConf privacy mode fetch causing ~188s T2 init delay#303
tabbas651 wants to merge 10 commits intodevelopfrom
topic/RDKEMW-15007

Conversation

@tabbas651
Copy link
Copy Markdown
Contributor

@tabbas651 tabbas651 commented Mar 27, 2026

Reason for change: When UserSettings sets privacy mode to DO_NOT_SHARE during boot,
the rbus SET handler (t2PropertyDataSetHandler) runs synchronously
on the single rbus callback thread — executing deleteAllProfiles()
with pthread_join, disk I/O, and XConf client restart.

The restarted XConf thread calls appendRequestParams() which uses
getParameterValue(PRIVACYMODES_RFC) to fetch privacy mode. This
issues rbus_get() on T2's own bus handle, requiring the same rbus
callback thread that is already blocked processing the SET handler.
This creates a self-deadlock with a 15s rbus timeout per attempt,
compounded by XCONF_RETRY_TIMEOUT (180s), stalling T2 initialization.

Additionally, privacyModeVal global pointer has no mutex protection,
allowing use-after-free and NULL dereference under concurrent
SET/GET operations during stress scenarios.

Cascading effect: While T2 is stalled, external callers (e.g.
WPEFramework SystemServices plugin) invoking t2_event_s() block
15s each on rbus_getUint(Telemetry.OperationalStatus), delaying
the dispatch thread by ~188s and blocking sendNotify() for
EVT_ONSYSTEMPOWERSTATECHANGED. Under concurrent privacy mode
toggling, PROVIDER_NOT_RESPONDING with 26+ messages queued and
dropped, telemetry events fall back to file caching.

Fix:

  • rbusInterface.c: Add privacyModeMutex (PTHREAD_MUTEX_INITIALIZER)
    to serialize all reads/writes of privacyModeVal in SET and GET
    handlers, preventing use-after-free and NULL dereference.

  • rbusInterface.c: Add privacyModeCallbackWorker() detached thread.
    The SET handler now validates input, updates privacyModeVal under
    mutex, calls setPrivacyMode() for persistence, then dispatches
    heavy callbacks (mprofilesDeleteCallBack, privacyModesDoNotShareCallBack)
    to the worker thread. RBUS handler returns immediately, preventing
    handler thread starvation and cascading 180s timeout loops.

  • xconfclient.c: Replace getParameterValue(PRIVACYMODES_RFC) with
    direct getPrivacyMode() call from privacycontrol library. The
    platform implementation (provided via meta layer bbappend) reads
    from a local in-memory cache (PRIVACY_STATE), with fallback to
    Thunder JSON-RPC (local HTTP) and persistent storage — none of
    which use rbus, eliminating the self-deadlock.

  • xconf-client/Makefile.am: Add privacycontrol include path and
    build dependency when IS_PRIVACYCONTROL_ENABLED is set. Add else
    branch for libxconfclient_la_CFLAGS to define the variable in all
    automake conditions, fixing autoreconf failure.

  • persistence.c: Change rm -f to rm -rf in clearPersistenceFolder()
    non-LIBSYSWRAPPER path. PRIVACYMODE_PATH is a directory, not a
    file; rm -f fails silently leaving stale persistence data.

Test Procedure: Concurrent stress test with multiple privacy mode toggles
(SHARE/DO_NOT_SHARE via curl JSON-RPC), telemetry marker submissions
(telemetry2_0_client), rbuscli reads, and SIGUSR1 signal — all
running simultaneously in background.
Risks: Medium
Signed-off-by: Thamim Razith ThamimRazith_AbbasAli@comcast.com

@tabbas651 tabbas651 requested a review from a team as a code owner March 27, 2026 23:54
Copilot AI review requested due to automatic review settings March 27, 2026 23:54
@tabbas651 tabbas651 force-pushed the topic/RDKEMW-15007 branch from b2370a0 to c111e67 Compare March 27, 2026 23:57
…Conf privacy mode fetch causing ~188s T2 init delay

Reason for change: When UserSettings sets privacy mode to DO_NOT_SHARE during boot,
the rbus SET handler (t2PropertyDataSetHandler) runs synchronously
on the single rbus callback thread — executing deleteAllProfiles()
with pthread_join, disk I/O, and XConf client restart.

The restarted XConf thread calls appendRequestParams() which uses
getParameterValue(PRIVACYMODES_RFC) to fetch privacy mode. This
issues rbus_get() on T2's own bus handle, requiring the same rbus
callback thread that is already blocked processing the SET handler.
This creates a self-deadlock with a 15s rbus timeout per attempt,
compounded by XCONF_RETRY_TIMEOUT (180s), stalling T2 initialization.

Cascading effect: While T2 is stalled, external callers (e.g.
WPEFramework SystemServices plugin) invoking t2_event_s() block
15s each on rbus_getUint(Telemetry.OperationalStatus), delaying
the dispatch thread by ~188s and blocking sendNotify() for
EVT_ONSYSTEMPOWERSTATECHANGED.

Fix:
- xconfclient.c: Replace getParameterValue(PRIVACYMODES_RFC) with
  direct getPrivacyMode() call from privacycontrol library. The
  platform implementation (provided via meta layer bbappend) reads
  from a local in-memory cache (PRIVACY_STATE), with fallback to
  Thunder JSON-RPC (local HTTP) and persistent storage — none of
  which use rbus, eliminating the self-deadlock.
`
- xconf-client/Makefile.am: Add privacycontrol include path and
  build dependency when IS_PRIVACYCONTROL_ENABLED is set.

- telemetry_busmessage_sender.c: Add fast-fail file marker check
  (T2_COMPONENT_READY) before rbus_getUint() in isCachingRequired()
  to avoid 15s blocking timeout when T2 is not yet ready.
Test Procedure: please refer the ticket comments
Risks: Medium
Signed-off-by: Thamim Razith <ThamimRazith_AbbasAli@comcast.com>
@tabbas651 tabbas651 force-pushed the topic/RDKEMW-15007 branch from c111e67 to f55c6e9 Compare March 27, 2026 23:58
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Addresses a startup self-deadlock and knock-on event dispatch delays by removing an rbus re-entrancy path during XConf privacy-mode fetch and by avoiding blocking rbus readiness queries before T2 is initialized.

Changes:

  • xconfclient.c: Switch privacy mode query parameter generation to use getPrivacyMode() (privacycontrol) instead of getParameterValue(PRIVACYMODES_RFC) (rbus).
  • xconf-client/Makefile.am: Add conditional include path/build dependency for privacycontrol when enabled.
  • telemetry_busmessage_sender.c: Fast-fail to caching if the /tmp/.t2ReadyToReceiveEvents marker is absent, avoiding 15s rbus timeouts during early boot.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
source/xconf-client/xconfclient.c Replaces rbus-based privacy mode fetch with privacycontrol API for XConf request parameterization.
source/xconf-client/Makefile.am Adds privacycontrol include/dependency under IS_PRIVACYCONTROL_ENABLED.
source/commonlib/telemetry_busmessage_sender.c Adds marker-file precheck to avoid blocking rbus calls when T2 isn’t ready.
Comments suppressed due to low confidence (1)

source/xconf-client/Makefile.am:49

  • When IS_PRIVACYCONTROL_ENABLED is set, this library now calls getPrivacyMode but only adds the privacycontrol library to libxconfclient_la_DEPENDENCIES. DEPENDENCIES controls build order but doesn’t ensure the symbol is linked/propagated to downstream link steps. Add the privacycontrol .la to libxconfclient_la_LIBADD (or appropriate LDADD/LIBADD) so consumers of libxconfclient.la reliably get the required link dependency.
libxconfclient_la_CFLAGS += $(PRIVACYCONTROL_FLAG)
libxconfclient_la_CPPFLAGS += -I${top_srcdir}/source/privacycontrol
libxconfclient_la_DEPENDENCIES += ${top_builddir}/source/privacycontrol/libt2thunder_privacycontrol.la
endif

…tion and race condition on privacyModeVal

Reason for change: When UserSettings sets privacy mode to DO_NOT_SHARE during boot,
the rbus SET handler (t2PropertyDataSetHandler) runs synchronously
on the single rbus callback thread — executing deleteAllProfiles()
with pthread_join, disk I/O, and XConf client restart.

The restarted XConf thread calls appendRequestParams() which uses
getParameterValue(PRIVACYMODES_RFC) to fetch privacy mode. This
issues rbus_get() on T2's own bus handle, requiring the same rbus
callback thread that is already blocked processing the SET handler.
This creates a self-deadlock with a 15s rbus timeout per attempt,
compounded by XCONF_RETRY_TIMEOUT (180s), stalling T2 initialization.

Additionally, privacyModeVal global pointer has no mutex protection,
allowing use-after-free and NULL dereference under concurrent
SET/GET operations during stress scenarios.

Cascading effect: While T2 is stalled, external callers (e.g.
WPEFramework SystemServices plugin) invoking t2_event_s() block
15s each on rbus_getUint(Telemetry.OperationalStatus), delaying
the dispatch thread by ~188s and blocking sendNotify() for
EVT_ONSYSTEMPOWERSTATECHANGED. Under concurrent privacy mode
toggling, PROVIDER_NOT_RESPONDING with 26+ messages queued and
dropped, telemetry events fall back to file caching.

Fix:
- rbusInterface.c: Add privacyModeMutex (PTHREAD_MUTEX_INITIALIZER)
  to serialize all reads/writes of privacyModeVal in SET and GET
  handlers, preventing use-after-free and NULL dereference.

- rbusInterface.c: Add privacyModeCallbackWorker() detached thread.
  The SET handler now validates input, updates privacyModeVal under
  mutex, calls setPrivacyMode() for persistence, then dispatches
  heavy callbacks (mprofilesDeleteCallBack, privacyModesDoNotShareCallBack)
  to the worker thread. RBUS handler returns immediately, preventing
  handler thread starvation and cascading 180s timeout loops.

- xconfclient.c: Replace getParameterValue(PRIVACYMODES_RFC) with
  direct getPrivacyMode() call from privacycontrol library. The
  platform implementation (provided via meta layer bbappend) reads
  from a local in-memory cache (PRIVACY_STATE), with fallback to
  Thunder JSON-RPC (local HTTP) and persistent storage — none of
  which use rbus, eliminating the self-deadlock.

- xconf-client/Makefile.am: Add privacycontrol include path and
  build dependency when IS_PRIVACYCONTROL_ENABLED is set. Add else
  branch for libxconfclient_la_CFLAGS to define the variable in all
  automake conditions, fixing autoreconf failure.

- telemetry_busmessage_sender.c: Add fast-fail file marker check
  (T2_COMPONENT_READY) before rbus_getUint() in isCachingRequired()
  to avoid blocking on rbus timeout when T2 is not ready.

- persistence.c: Change rm -f to rm -rf in clearPersistenceFolder()
  non-LIBSYSWRAPPER path. PRIVACYMODE_PATH is a directory, not a
  file; rm -f fails silently leaving stale persistence data.

Test Procedure: Concurrent stress test with multiple privacy mode toggles
(SHARE/DO_NOT_SHARE via curl JSON-RPC), telemetry marker submissions
(telemetry2_0_client), rbuscli reads, and SIGUSR1 signal — all
running simultaneously in background.
Risks: Medium
Signed-off-by: Thamim Razith <ThamimRazith_AbbasAli@comcast.com>
Copilot AI review requested due to automatic review settings March 30, 2026 04:24
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Copilot AI review requested due to automatic review settings March 30, 2026 18:28
@tabbas651 tabbas651 changed the title RDKEMW-15927 ,RDKEMW-15007 : [APACA/Xione DE] rbus self-deadlock in XConf privacy mode fetch causing ~188s T2 init delay RDKEMW-15927 : [Xione DE] rbus self-deadlock in XConf privacy mode fetch causing ~188s T2 init delay Mar 30, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

Comment on lines 614 to 618
char* data = rbusValue_ToString(paramValue_t, NULL, 0);
if(privacyModeVal != NULL)
{
free(privacyModeVal);
privacyModeVal = NULL;
}
if((strcmp(data, "SHARE") != 0) && (strcmp(data, "DO_NOT_SHARE") != 0))
{
T2Info("Unexpected privacy Mode value %s\n", data);
free(data);
Copy link

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

t2PropertyDataSetHandler() uses the result of rbusValue_ToString() without checking for NULL. If rbusValue_ToString() fails, the subsequent strcmp() calls will crash. Add a NULL check for data and return RBUS_ERROR_INVALID_INPUT (or similar) when conversion fails.

Copilot uses AI. Check for mistakes.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 30, 2026 18:36
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

shibu-kv
shibu-kv previously approved these changes Mar 30, 2026
Copilot AI review requested due to automatic review settings March 30, 2026 19:42
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

…etString after ownership transfer

Agent-Logs-Url: https://github.com/rdkcentral/telemetry/sessions/d5b743ff-3289-4141-b51a-399971064c4c

Co-authored-by: shibu-kv <89052442+shibu-kv@users.noreply.github.com>
Copilot AI requested a review from shibu-kv March 30, 2026 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants