Python 3.13, linting/quality, and agent reliability

- Raise requires-python to >=3.13 across all projects/libs; set ruff target to py313
- Fix bare excepts, unused variables/imports, Optional→union syntax, and import ordering
- Refactor chatbot MCP server to async subprocess with health checks and graceful shutdown
- Simplify file-enrichment prod Dockerfile to use uv sync with UV_PROJECT_ENVIRONMENT
- Fix module_loader to use fully qualified analyzer module name
- Make YARA _analyze_yara async and fix rule_manager access in workflow
- Enable LiteLLM healthcheck and add agents→litellm dependency
- Add PE analyzer tests with PyInstaller fixture;
- Update PDF tests for nested metadata
- Suppress third-party DeprecationWarnings, use async-lru for LLM status cache
- Add PHOENIX_ENABLED validation to nemesis-ctl.sh; remove debug pprint in agents
- Add Claude Code managing-packages skill, update CLAUDE.md guidance

Author: Lee Chagolla-Christensen

.claude/skills/managing-packages.md

@@ -0,0 +1,63 @@
+---
+name: managing-packages
+description: Manages Python package dependencies. Use when adding, upgrading, removing, or syncing Python/pypi packages in projects or libs.
+---
+
+# Managing Packages with uv
+
+## Commands
+
+All commands must be run from the specific project/library directory.
+
+### Add a package
+
+```bash
+cd /home/itadmin/code/Nemesis/{path} && uv add {package}
+```
+
+### Add dev dependency
+
+```bash
+cd /home/itadmin/code/Nemesis/{path} && uv add --dev {package}
+```
+
+### Upgrade a package
+
+```bash
+cd /home/itadmin/code/Nemesis/{path} && uv add {package}@latest
+```
+
+### Remove a package
+
+```bash
+cd /home/itadmin/code/Nemesis/{path} && uv remove {package}
+```
+
+### Sync dependencies
+
+```bash
+cd /home/itadmin/code/Nemesis/{path} && uv sync
+```
+
+## Project paths
+
+```
+projects/web_api/
+projects/file_enrichment/
+projects/document_conversion/
+projects/cli/
+projects/alerting/
+projects/housekeeping/
+projects/agents/
+libs/common/
+libs/file_enrichment_modules/
+libs/chromium/
+libs/file_linking/
+libs/nemesis_dpapi/
+```
+
+## Notes
+
+- Always use `uv`, never `pip`
+- Commit both `pyproject.toml` and `uv.lock`
+- Use `uv run` to execute commands in the project's environment

CLAUDE.md

@@ -48,17 +48,21 @@ Nemesis is an open-source, centralized data processing platform (v2.0) that inge
 cd projects/web_api && uv sync
 ```
 
-### Linting & Formatting (Ruff)
+### Package Management
+
+For adding, upgrading, or removing packages, use the `/managing-packages` skill.
 
+### Linting & Formatting (Ruff)
+Run linting/formatting after each fix/change.
 ```bash
 # Check all Python code (configured in root pyproject.toml)
-ruff check .
+uv run ruff check . --fix
 
 # Format code
-ruff format .
+uv run ruff format .
 
 # Check specific project
-cd projects/web_api && uv run ruff check .
+cd projects/web_api && uv run ruff check . --fix
 ```
 
 ### Testing
@@ -75,22 +79,27 @@ cd projects/web_api && uv run pytest tests/test_file.py::test_function_name
 ```
 
 ### Docker Commands
+Information about building dev/prod container images can be found in the [docker compose docs](./docs/docker_compose.md).
 
+General instructions for dev docker containers/images:
 ```bash
 # Build base images first (required before building services)
 docker compose -f compose.base.yaml build
 
 # View logs for a service
 docker compose logs -f web-api
 
-# Rebuild and restart single service
-docker compose up -d --build web-api
+# Rebuild Dapr-enabled services (must include the -dapr sidecar)
+docker compose up -d --build file-enrichment file-enrichment-dapr
+docker compose up -d --build web-api web-api-dapr
 ```
 
+**Note:** Services using Dapr (file-enrichment, web-api, alerting, etc.) have a companion `-dapr` sidecar container. When rebuilding these services, always restart both the service and its Dapr sidecar to ensure proper communication.
+
 ## Architecture
 
 ### Tech Stack
-- **Python 3.12-3.13** with uv for dependency management
+- **Python 3.13** with uv for dependency management
 - **FastAPI** for REST services
 - **React 18 + Vite + TypeScript** for frontend
 - **PostgreSQL** for database

compose.yaml

@@ -238,9 +238,9 @@ services:
       interval: 5s
       timeout: 5s
       retries: 5
-    volumes: []
-      # # Uncomment the following line to use custom YARA rules
-      # - ./libs/file_enrichment_modules/yara_rules/prod/:/yara_rules/:ro
+    volumes:
+      # Uncomment the following line to use custom YARA rules
+      - ./libs/file_enrichment_modules/yara_rules/prod/:/yara_rules/:ro
     environment: &file-enrichment-environment
       ENABLE_PII_DETECTION: ${ENABLE_PII_DETECTION:-false}
       PII_DETECTION_THRESHOLD: ${PII_DETECTION_THRESHOLD:-0.7}
@@ -1083,6 +1083,7 @@ services:
       placement: { condition: service_started }
       rabbitmq: { condition: service_healthy }
       hasura: { condition: service_healthy }
+      litellm: { condition: service_healthy }
     healthcheck: *healthcheck-python-svc
     labels:
       - "traefik.enable=true"
@@ -1163,20 +1164,13 @@ services:
     healthcheck:
       test:
         [
-          "CMD",
-          "wget",
-          "--quiet",
-          "--tries=1",
-          "-O",
-          "/dev/null",
-          "--header=Authorization: Bearer sk-${LITELLM_ADMIN_PASSWORD:-admin123}",
-          "http://localhost:4000/health",
+          "CMD-SHELL",
+          "python3 -c \"import urllib.request,json; r=urllib.request.urlopen(urllib.request.Request('http://localhost:4000/health',headers={'Authorization':'Bearer sk-${LITELLM_ADMIN_PASSWORD:-admin123}'})); d=json.load(r); exit(0 if d.get('healthy_count',0)>0 and d.get('unhealthy_count',0)==0 else 1)\"",
         ]
       interval: 30s
       timeout: 10s
       retries: 3
-      start_period: 30s
-      disable: true
+      start_period: 60s
     logging: *logging-config
     labels:
       - "traefik.enable=true"

docs/agents.md

@@ -92,13 +92,13 @@ The Chatbot agent powers the "Chatbot" icon in the left navigation panel. This a
 
 ## The Nemesis Web Interface
 
-### Monitoring
+### Phoenix LLM Tracing
 
-If the `--monitoring` flag is also passed to the `./tools/nemesis-ctl.sh` script, the [Arize Phoenix](https://github.com/Arize-ai/phoenix) will be deployed to allow tracking of the inputs/outputs sent to the LLM (at the /phoenix route):
+When the `--llm` flag is passed to `./tools/nemesis-ctl.sh`, [Arize Phoenix](https://github.com/Arize-ai/phoenix) is deployed (available at the /phoenix route) to allow tracking of the inputs/outputs sent to the LLM.
 
 ![Arize Phoenix Tracing](images/arize_phoenix_tracing.png)
 
-If the monitoring profile is used and Arize Phoenix is deployed, the Nemesis frontend will dynamically display links to the Phoenix interface in the Help menu as well.
+When Arize Phoenix is deployed, the Nemesis frontend will dynamically display links to the Phoenix interface in the Help menu.
 
 ![Nemesis Dynamic Help Menu](images/nemesis_dynamic_help_menu.png)
 

infra/litellm/config.yml

@@ -6,6 +6,7 @@ model_list:
       aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
       aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
       aws_region_name: "us-east-1"
+      
   # # used to triage findings and determine their validity
   # - model_name: triage
   #   litellm_params:

libs/chromium/chromium/chromekey.py

@@ -53,9 +53,9 @@ async def retry_decrypt_chrome_key(chrome_key_id: int, dpapi_manager: DpapiManag
             logger.warning("Chrome key not found", chrome_key_id=chrome_key_id)
             return result
 
-        record_id = row["id"]
+        _record_id = row["id"]  # noqa: F841
         source = row["source"]
-        key_masterkey_guid = row["key_masterkey_guid"]
+        _key_masterkey_guid = row["key_masterkey_guid"]  # noqa: F841
         key_bytes_enc = row["key_bytes_enc"]
         key_bytes_dec = row["key_bytes_dec"]
         key_is_decrypted = row["key_is_decrypted"]

libs/chromium/chromium/cookies.py

@@ -140,7 +140,7 @@ async def _insert_cookies(
                     if value_dec_bytes:
                         value_dec = value_dec_bytes.decode("utf-8", errors="replace")
                         is_decrypted = True
-                except:
+                except Exception:
                     pass
 
             # Get state key ID for key/abe encryption

libs/chromium/chromium/local_state.py

@@ -489,16 +489,16 @@ async def retry_decrypt_state_key(state_key_id: int, dpapi_manager: DpapiManager
             logger.warning("State key not found", state_key_id=state_key_id)
             return result
 
-        record_id = row["id"]
+        _record_id = row["id"]  # noqa: F841
         source = row["source"]
         username = row["username"]
         browser = row["browser"]
-        key_masterkey_guid = row["key_masterkey_guid"]
+        _key_masterkey_guid = row["key_masterkey_guid"]  # noqa: F841
         key_bytes_enc = row["key_bytes_enc"]
         key_bytes_dec = row["key_bytes_dec"]
         key_is_decrypted = row["key_is_decrypted"]
         app_bound_key_enc = row["app_bound_key_enc"]
-        app_bound_key_system_masterkey_guid = row["app_bound_key_system_masterkey_guid"]
+        _app_bound_key_system_masterkey_guid = row["app_bound_key_system_masterkey_guid"]  # noqa: F841
         app_bound_key_user_masterkey_guid = row["app_bound_key_user_masterkey_guid"]
         app_bound_key_system_dec = row["app_bound_key_system_dec"]
         app_bound_key_user_dec = row["app_bound_key_user_dec"]
@@ -880,7 +880,7 @@ async def retry_decrypt_state_keys_for_chromekey(source: str, chromekey: bytes,
             username = row["username"]
             browser = row["browser"]
             app_bound_key_user_dec = row["app_bound_key_user_dec"]
-            user_masterkey_guid = row["app_bound_key_user_masterkey_guid"]
+            _user_masterkey_guid = row["app_bound_key_user_masterkey_guid"]  # noqa: F841
 
             result["state_keys_attempted"] += 1
 

libs/chromium/chromium/logins.py

@@ -124,7 +124,7 @@ async def _insert_logins(
                     if password_dec_bytes:
                         password_value_dec = password_dec_bytes.decode("utf-8", errors="replace")
                         is_decrypted = True
-                except:
+                except Exception:
                     pass
 
             # Get state key ID for key/abe encryption

libs/chromium/pyproject.toml

@@ -4,7 +4,7 @@ version = "0.1.0"
 description = "Modules Nemesis uses to handle Chromium files"
 authors = [{name = "SpecterOps"}]
 readme = "README.md"
-requires-python = ">=3.12,<3.14"
+requires-python = ">=3.13,<3.14"
 
 dependencies = [
     "impacket>=0.13.0,<0.14.0",