update user groups issue

Author: a-happysoft

.gitignore

@@ -148,3 +148,4 @@ Taskfile.yml
 /.logs
 /logs
 webapp_logs.zip
+/.schemas

docs/user_groups_issue.md

@@ -0,0 +1,189 @@
+# User Groups Discrepancy Issue
+
+## Summary
+
+We've identified a discrepancy between the database state and API responses when retrieving user groups. When a user requests their groups, they're not receiving all groups where they are members. Specifically, when user ID `1062` requests their groups, they should receive 3 groups (IDs 22, 25, and 28), but only receive 2 groups (IDs 25 and 28).
+
+## Investigation Details
+
+### Debug Logs
+
+The logs show only two groups being returned:
+
+```
+2025-05-21 21:38:17,604 - src.routers.user_groups - DEBUG - Fetching groups for user 1062...
+2025-05-21 21:38:17,604 - src.database.user_groups - DEBUG - Getting user groups with users for user_id: 1062
+2025-05-21 21:38:17,604 - src.database.user_groups - DEBUG - Fetching groups where user 1062 is a member...
+2025-05-21 21:38:18,656 - src.database.user_groups - DEBUG - Found 1 groups where user 1062 is a member: [28]
+2025-05-21 21:38:18,656 - src.database.user_groups - DEBUG - Fetching groups where user 1062 is an admin...
+2025-05-21 21:38:19,606 - src.database.user_groups - DEBUG - Found 1 groups where user 1062 is an admin: [25]
+2025-05-21 21:38:19,606 - src.database.user_groups - DEBUG - Added 1 additional groups as admin (not already as member): [25]
+2025-05-21 21:38:19,606 - src.database.user_groups - DEBUG - Total: Found 2 user groups with users for user_id: 1062, Group IDs: [25, 28]
+```
+
+### Database State
+
+Our database queries confirm that the user should be in 3 groups:
+
+```sql
+SELECT id, name, user_ids, admin_ids 
+FROM user_groups 
+WHERE 1062 = ANY(user_ids) OR 1062 = ANY(admin_ids) 
+ORDER BY id;
+```
+
+Result:
+```
+ id |    name     |    user_ids     | admin_ids 
+----+-------------+-----------------+-----------
+ 22 | UNDP Studio | {1062,774,1067} | {}
+ 25 | GROUP 3     | {1062}          | {1062}
+ 28 | test4       | {774,1062}      | {774}
+```
+
+### Code Analysis
+
+The code in `user_groups.py` uses the correct SQL syntax to retrieve groups where a user is a member or admin:
+
+1. First query: `SELECT ... FROM user_groups WHERE %s = ANY(user_ids) ORDER BY created_at DESC;`
+2. Second query: `SELECT ... FROM user_groups WHERE %s = ANY(admin_ids) ORDER BY created_at DESC;`
+
+These queries should correctly find all groups, but the first query is only returning group 28, not both 22 and 28 as expected.
+
+## Impact
+
+Users may not see all groups they belong to in the application, which could lead to:
+
+1. Reduced access to signals shared in "missing" groups
+2. Confusion about group membership
+3. Workflow disruptions if users expect to find signals in specific groups
+
+## Possible Causes
+
+1. SQL query execution issues
+2. Application-level filtering not visible in the code
+3. A caching or stale data issue
+4. Transaction isolation level issues
+5. Potential race condition if groups are being modified simultaneously
+
+## Fix Implemented
+
+We've implemented a comprehensive solution with multiple layers of improvements:
+
+### 1. Enhanced Primary Functions
+
+Modified the approach in the affected functions to use a single combined query with explicit array handling:
+
+```python
+# Run a direct SQL query to ensure array type handling is consistent
+query = """
+SELECT 
+    id, 
+    name,
+    signal_ids,
+    user_ids,
+    admin_ids,
+    collaborator_map,
+    created_at
+FROM 
+    user_groups
+WHERE 
+    %s = ANY(user_ids) OR %s = ANY(admin_ids)
+ORDER BY 
+    created_at DESC;
+"""
+
+await cursor.execute(query, (user_id, user_id))
+```
+
+We also added explicit type conversion when checking for user membership:
+
+```python
+# Track membership rigorously by explicitly converting IDs to integers
+is_member = False
+if data['user_ids']:
+    is_member = user_id in [int(uid) for uid in data['user_ids']]
+    
+is_admin = False
+if data['admin_ids']:
+    is_admin = user_id in [int(aid) for aid in data['admin_ids']]
+```
+
+### 2. Debug Functions
+
+Added a `debug_user_groups` function that runs multiple direct queries to diagnose issues:
+
+```python
+async def debug_user_groups(cursor: AsyncCursor, user_id: int) -> dict:
+    # Various direct SQL queries to check database state 
+    # and array position functions
+    # ...
+```
+
+### 3. Fallback Implementation
+
+Created a completely separate direct SQL implementation in `user_groups_direct.py` as a fallback:
+
+```python
+async def get_user_groups_direct(cursor: AsyncCursor, user_id: int) -> List[UserGroup]:
+    """
+    Get all groups that a user is a member of or an admin of using direct SQL.
+    """
+    # Simple, direct SQL with minimal processing
+    # ...
+```
+
+### 4. Automatic Fallback in API
+
+Modified the user groups router to automatically detect and handle discrepancies:
+
+```python
+# Check if there's a mismatch between direct query and regular function
+if missing_ids:
+    logger.warning(f"MISMATCH! Direct query found groups {direct_group_ids} but function returned only {fetched_ids}")
+    logger.warning(f"Missing groups: {missing_ids}")
+    
+    # Fall back to direct SQL implementation
+    logger.warning("Falling back to direct SQL implementation")
+    user_groups = await user_groups_direct.get_user_groups_with_users_direct(cursor, user.id)
+    logger.info(f"Direct SQL implementation returned {len(user_groups)} groups")
+```
+
+These changes ensure:
+- More reliable querying of user group memberships
+- Better debug information if issues persist
+- Automatic fallback to a simpler implementation if needed
+- More detailed logging throughout the process
+
+After these changes, users should see all groups where they are members or admins consistently.
+
+## Additional Fix: Signal Entity can_edit Attribute
+
+During testing, we discovered that the Signal entity was missing the `can_edit` attribute that's dynamically added in user group contexts. This caused AttributeError exceptions when trying to access `signal.can_edit`.
+
+### Issue
+```python
+AttributeError: 'Signal' object has no attribute 'can_edit'
+```
+
+### Solution
+Added the `can_edit` field to the Signal entity definition:
+
+```python
+can_edit: bool = Field(
+    default=False,
+    description="Whether the current user can edit this signal (set dynamically based on group membership and collaboration).",
+)
+```
+
+This ensures that:
+- The Signal model accepts the `can_edit` attribute when created
+- The attribute defaults to `False` for signals that don't have edit permissions
+- Both the regular and direct SQL implementations can properly set this attribute
+- Frontend code can safely access `signal.can_edit` without errors
+
+The fix has been applied to both endpoints:
+1. `/user-groups/me` (user groups without signals)
+2. `/user-groups/me/with-signals` (user groups with signals)
+
+Both endpoints now include the same debug checks and automatic fallback to direct SQL if discrepancies are detected.

src/authentication.py

@@ -142,10 +142,10 @@ async def authenticate_user(
                 token = test_token
 
         # Default user data for local development
-        local_email = "test.user@undp.org"
-        name = "Test User"
-        unit = "Data Futures Exchange (DFx)"
-        acclab = False
+        local_email = os.environ.get("TEST_USER_EMAIL", "test.user@undp.org")
+        name = os.environ.get("TEST_USER_NAME", "Test User")
+        unit = os.environ.get("TEST_USER_UNIT", "Data Futures Exchange (DFx)")
+        acclab = os.environ.get("TEST_USER_ACCLAB", False)
 
         # Check for specific test tokens
         if token == "test-admin-token":

src/database/signals.py

@@ -92,6 +92,12 @@ async def search_signals(cursor: AsyncCursor, filters: SignalFilters) -> SignalP
              AND (%(score)s IS NULL OR score = %(score)s)
              AND (%(unit)s IS NULL OR unit_region = %(unit)s OR unit_name = %(unit)s)
              AND (%(query)s IS NULL OR text_search_field @@ websearch_to_tsquery('english', %(query)s))
+             AND (%(user_email)s IS NOT NULL AND (
+                  private = FALSE OR 
+                  created_by = %(user_email)s OR
+                  %(is_admin)s = TRUE OR
+                  %(is_staff)s = TRUE
+             ))
         ORDER BY
             {filters.order_by} {filters.direction}
         OFFSET
@@ -128,6 +134,7 @@ async def create_signal(cursor: AsyncCursor, signal: Signal, user_group_ids: Lis
         An ID of the signal in the database.
     """
     logger.info(f"Creating new signal with headline: '{signal.headline}', created by: {signal.created_by}")
+    logger.info(f"All Signal fields: {signal.model_dump()}")
     if user_group_ids:
         logger.info(f"Will add signal to user groups: {user_group_ids}")
 
@@ -152,7 +159,8 @@ async def create_signal(cursor: AsyncCursor, signal: Signal, user_group_ids: Lis
                 keywords,
                 location,
                 secondary_location,
-                score
+                score,
+                private
             )
             VALUES (
                 %(status)s,
@@ -172,7 +180,8 @@ async def create_signal(cursor: AsyncCursor, signal: Signal, user_group_ids: Lis
                 %(keywords)s,
                 %(location)s,
                 %(secondary_location)s,
-                %(score)s
+                %(score)s,
+                %(private)s
             )
             RETURNING
                 id
@@ -411,7 +420,8 @@ async def update_signal(cursor: AsyncCursor, signal: Signal, user_group_ids: Lis
                  keywords = COALESCE(%(keywords)s, keywords),
                  location = COALESCE(%(location)s, location),
                  secondary_location = COALESCE(%(secondary_location)s, secondary_location),
-                 score = COALESCE(%(score)s, score)
+                 score = COALESCE(%(score)s, score),
+                 private = COALESCE(%(private)s, private)
             WHERE
                 id = %(id)s
             RETURNING
@@ -559,6 +569,8 @@ async def read_user_signals(
     cursor: AsyncCursor,
     user_email: str,
     status: Status,
+    is_admin: bool = False,
+    is_staff: bool = False,
 ) -> list[Signal]:
     """
     Read signals from the database using a user email and status filter.
@@ -571,6 +583,10 @@ async def read_user_signals(
         An email of the user whose signals to read.
     status : Status
         A status of signals to filter by.
+    is_admin : bool, optional
+        Whether the user is an admin, by default False
+    is_staff : bool, optional
+        Whether the user is staff, by default False
 
     Returns
     -------
@@ -596,6 +612,9 @@ async def read_user_signals(
             created_by = %s AND status = %s
         ;
         """
+    # Since this function is explicitly for reading a user's OWN signals,
+    # we don't need additional private/public filtering here.
+    # The user is the creator, so they should see all their signals regardless of privacy setting.
     await cursor.execute(query, (user_email, status))
     rows = await cursor.fetchall()
     return [Signal(**row) for row in rows]