Memory not persisting across sessions in trusted-proxy auth mode

[bvaturi]

Summary

When running in trusted-proxy auth mode with OIDC, agent memory (pgvector) does not persist across sessions for the same user. Memory saved in session 1 cannot be found in session 2, even though both sessions belong to the same authenticated user.

Root Cause

The memory save and search operations use different user_id values:

• Save uses session.user_id which is set to system:serviceaccount:kagent:<agent-name> (the agent pod's service account identity)
• Search (in a new session) uses the authenticated user's ID from the JWT sub claim (e.g., dcfbb20c-ed01-4895-a922-cf80bd731bbe)

This mismatch means memories saved under the agent's SA identity can never be found when searching under the real user's identity.

Flow Analysis

1. User sends message via UI → oauth2-proxy injects JWT → controller authenticates user as dcfbb20c-...
2. Controller proxies A2A request to agent pod with X-User-Id: dcfbb20c-... header
3. Python A2A server's KAgentRequestContextBuilder reads x-user-id from context.state["headers"] and sets call_context.user
4. _get_user_id() in request_converter.py correctly returns the real user ID from call_context.user.user_name
5. However, the session events are posted to the controller API with user_id=system:serviceaccount:kagent:testme (the agent's own SA)
6. When SaveMemoryTool calls memory_service.add_memory(user_id=tool_context.session.user_id), it uses the session's user_id which is the agent's SA
7. When LoadMemoryTool searches in a new session, the new session correctly has the real user ID, but the stored memory has the agent's SA as user_id

Evidence from logs

# Save (session 1) - uses agent SA:
Successfully added memory ID e59f56cc-... for user system:serviceaccount:kagent:testme agent kagent__NS__testme

# Search (session 2) - uses real user ID, finds nothing:
HTTP Request: GET .../api/sessions/ctx-7b8f290c-...?user_id=dcfbb20c-ed01-4895-a922-cf80bd731bbe
No memories found for query: all saved memories

Expected Behavior

Memory should be scoped to the authenticated user (from OIDC JWT), not the agent's service account. A user should be able to save memory in one session and retrieve it in any future session with the same agent.

Proposed Fix

The session.user_id should be set to the authenticated user's ID (from x-user-id header / call_context.user.user_name) when the session is created, not the agent's service account. This likely requires changes in:

1. kagent/python/packages/kagent-adk/src/kagent/adk/_agent_executor.py — where the session is prepared/created, ensure the user_id comes from the authenticated user
2. The session creation flow should prioritize the propagated x-user-id over the agent's own identity

Environment

• kagent v0.9.0
• Auth mode: trusted-proxy with Keycloak OIDC
• Memory: pgvector with Bedrock Titan Embed v2
• Agent runtime: Python ADK

Workaround

Memory works within the same session (same chat window). Cross-session memory requires this fix.

[kinthaiofficial]

Classic identity mismatch bug — the user_id divergence between save and search is a common pitfall in multi-tenant agent memory systems.

From running persistent memory for 31 concurrent agents in a multi-tenant setup:

The root cause pattern: This happens whenever the identity used for memory operations comes from different sources (service account vs. OIDC user). The fix is straightforward — always use the authenticated user identity for memory operations, never the agent's service account. But the deeper architectural lesson is:

Memory should be scoped by a stable user identifier, not session/runtime identity. We use a three-level memory scope:

1. User-level — persists across all sessions for the same user. The user_id must be the OIDC subject claim (stable, unique per user), not the session ID or service account
2. Agent-level — persists across sessions for a specific agent serving a specific user. Scoped by (user_id, agent_id) tuple
3. Session-level — ephemeral working memory, scoped by session_id, auto-expires

Per-agent encryption prevents cross-user leakage: Even with correct user_id scoping, a defense-in-depth approach encrypts memories per-user. We use deterministic encryption on metadata (enabling search without decryption) with full AES on content. This means even a bug like this one (wrong user_id in search query) would return encrypted results that can't be decrypted by the wrong user.

Diagnostic pattern: For memory persistence bugs, always check three things:

1. What identifier is used at write time?
2. What identifier is used at read time?
3. Are they guaranteed to be the same for the same logical user?

If any of these differ across auth modes (direct vs. proxy vs. service account), you have this bug.

Architecture reference on multi-tenant memory isolation: https://blog.kinthai.ai/openclaw-multi-tenancy-why-vm-per-user-doesnt-scale
Deep-dive on persistent memory: https://blog.kinthai.ai/why-character-ai-forgets-you-persistent-memory-architecture