Merge remote-tracking branch 'origin/main' into test-ui-fixes

Author: Bojan131

.codex/review-prompt.md

@@ -9,7 +9,7 @@ Read these files before reviewing:
 1. **`pr-diff.patch`** — The PR diff (generated at runtime). This is the primary input.
 2. **`AGENTS.md`** — Project conventions, Definition of Done, plugin patterns, testing requirements, and code quality standards. This is the source of truth for how code in this project should look.
 
-You may read other files in the repository to understand surrounding context (e.g., how a modified function is called, what a referenced constant is). However, all review comments must target lines that appear in the diff.
+You may read other files in the repository **only** to understand how code changed in the diff is called or referenced. Do not review, comment on, or mention code in files or packages that are not part of the diff. All review comments and the summary must be strictly scoped to changes introduced by this PR's diff — nothing else.
 
 ## Review Philosophy
 
@@ -26,19 +26,21 @@ When both exist, report blockers first.
 
 Do three passes:
 
-1. **Context + risk-map pass (mandatory)** — Read full touched files (not only hunk lines) and identify high-risk zones: platform/runtime boundaries, event handlers, async cleanup (`finally`), auth/validation boundaries, and repeated predicates/guards.
+1. **Context + risk-map pass (mandatory)** — Start from diff hunks, then read surrounding or full touched files when needed to evaluate maintainability, coupling, naming, and extraction opportunities. Use this context to assess changed behavior, not to run unrelated file-wide audits.
 2. **Blockers pass** — Scan for correctness bugs, security issues, API/schema contract breaks, missing migrations, data integrity risks, and missing tests for changed behavior. These are `🔴 Bug` comments.
 3. **Maintainability pass** — Scan for code bloat, readability issues, naming problems, pattern violations, hardcoded values, and architecture drift in touched areas. These are `🟡 Issue`, `🔵 Nit`, or `💡 Suggestion` comments.
 
 ### Comment Gate
 
-Before posting any comment, verify all three conditions:
+Before posting any comment, verify all four conditions:
 
 1. **Introduced by this diff** — The issue is introduced or materially worsened by the changes in this PR, not pre-existing.
 2. **Materially impactful** — The issue affects correctness, security, readability, or maintainability in a meaningful way. Not a theoretical concern.
 3. **Concrete fix direction** — You can suggest a specific fix or clear direction. If you can only say "this seems off" without a concrete suggestion, do not comment.
+4. **Scope fit** — If the issue is mainly in pre-existing code, the PR must touch the same function/module and fixing it must directly simplify, de-risk, or de-duplicate the new/changed code.
 
 If any check fails, skip the comment.
+Every comment must be traceable to changed behavior in this PR and anchored to a right-side line present in `pr-diff.patch`. Prefer added/modified lines; use nearby unchanged hunk lines only when necessary to explain a directly related issue.
 
 **Uncertainty guard:** If you are not certain an issue is real and cannot verify it from the diff and allowed context, do not label it `🔴 Bug`. Downgrade to `🟡 Issue` or `💡 Suggestion`, or skip it entirely.
 
@@ -53,29 +55,34 @@ If any check fails, skip the comment.
 - Logic errors, off-by-one, null/undefined handling, incorrect assumptions, race conditions.
 - Boundary conditions — empty arrays, null inputs, zero values, maximum values.
 - Error handling — swallowed errors, missing error propagation, unhelpful error messages. Do not flag missing error handling for internal code that cannot reasonably fail.
+- Streaming/multipart handlers — verify a request cannot send multiple responses (e.g., multi-file parts triggering repeated `res.json()` calls). If a route expects one file, ensure parser limits and single-response guards exist.
 - Unsafe runtime assumptions hidden by type assertions (`as ...`, `as any`, non-null `!`) when values come from events, external I/O, or platform-specific APIs.
 - Platform/runtime compatibility assumptions — usage of globals/APIs (`window`, `document`, `Node`, `process`, browser-only APIs) in cross-runtime code paths must be guarded.
 
 #### Security
 
+
 - Injection risks (SQL, command, XSS) when handling user input.
 - Hardcoded secrets — API keys, passwords, tokens in code.
 - Missing input validation at system boundaries (user input, external APIs). Not for internal function calls.
 - Auth bypass, privilege escalation, or missing authorization checks.
+- Filesystem path confinement — when IDs/paths come from requests, verify storage layers enforce root containment via resolved-path checks; do not rely only on caller-side sanitization.
 
 #### API Compatibility
 
 - Breaking changes to API response schemas or status codes without migration path.
 - Removed or renamed API endpoints, query parameters, or response fields that existing consumers depend on.
 - Database schema changes that require migration or backfill.
 - MCP tool signature changes (renamed tools, changed input schemas) that break existing clients.
+- HTTP status semantics — ensure client/input errors are 4xx and unexpected internal failures are 5xx; blanket 400 handling in catch-all paths is a correctness/API contract issue.
 
 #### Tests for Changed Behavior
 
 - New behavior must have corresponding tests covering core functionality and error handling.
 - Bug fixes must include a regression test that would have caught the original bug.
 - Changed behavior must have updated tests reflecting the new expectations.
 - If tests are present but brittle (testing implementation details rather than behavior), flag it.
+- For single-file upload endpoints, look for regression coverage of multi-file/malformed multipart inputs and confirm no double-response behavior.
 - Prefer tests that validate production behavior directly. If a test re-implements production decision logic locally, and could stay green while runtime behavior regresses, flag it and suggest importing shared runtime logic or testing via a higher-level behavior path.
 
 Missing tests for changed behavior are blockers (`🔴 Bug`) only when the change affects user-facing behavior, API contracts, or data integrity. Missing tests for internal refactors or trivial changes are `🟡 Issue`.
@@ -113,6 +120,7 @@ Missing tests for changed behavior are blockers (`🔴 Bug`) only when the chang
 - **Wrong import paths in tests** — Tests importing from `src/` instead of `dist/`.
 - **Missing test categories** — Tests without "Core Functionality" and "Error Handling" describe blocks.
 - **Mixing concerns** — Route handlers doing business logic, database queries in API handlers, etc.
+- **Cross-provider behavior drift** — When multiple providers/implementations exist, verify shared options and output semantics behave consistently unless explicitly documented otherwise.
 
 #### Hardcoded Values and Magic Constants
 
@@ -135,7 +143,9 @@ Do not flag one-off numeric literals that are self-explanatory in context (e.g.,
 - Type annotations for code that already type-checks.
 - Things that are clearly intentional design choices backed by existing patterns.
 - Pre-existing issues in unchanged code outside the diff.
+- Pre-existing issues in touched files when the PR does not introduce/worsen them.
 - Adding documentation unless a public API is clearly undocumented.
+- Repository-wide or file-wide audits not required by the changed behavior.
 
 ## Comment Format
 
@@ -174,4 +184,4 @@ The `line` field must refer to the line number in the new version of the file (r
 
 ## Summary
 
-Write a brief (2–4 sentence) overall assessment in the `summary` field. Lead with blockers if any exist. Mention whether the PR is clean/minimal or has code quality issues. Include one sentence on maintainability direction in touched areas (improved / neutral / worsened, and why). If the PR looks good, say so.
+Write a brief (2–4 sentence) overall assessment in the `summary` field covering **only** what this PR's diff changes. Do not mention code, packages, or behavior outside the diff. Lead with blockers if any exist. Mention whether the PR is clean/minimal or has code quality issues. Include one sentence on maintainability direction in touched areas (improved / neutral / worsened, and why). If the PR looks good, say so.

.github/workflows/codex-review.yml

@@ -75,7 +75,8 @@ jobs:
               }
             );
 
-            // Build set of valid (path:line) pairs from diff patches
+            // Build set of valid (path:line) pairs from right-side diff hunk lines
+            // (added + context). This keeps comments bound to changed areas.
             const validLines = new Set();
             for (const file of files) {
               // Skip binary/large/truncated files with no patch
@@ -89,20 +90,26 @@ jobs:
                   currentLine = parseInt(hunkMatch[1], 10);
                   continue;
                 }
-                // Deleted lines don't exist in the new file
-                if (line.startsWith('-')) continue;
-                // Context lines and added lines are valid targets
-                if (!line.startsWith('\\')) {
+                // Added lines are valid comment targets
+                if (line.startsWith('+')) {
                   validLines.add(`${file.filename}:${currentLine}`);
                   currentLine++;
+                  continue;
                 }
+                // Deleted lines don't exist in the new file
+                if (line.startsWith('-')) continue;
+                // Ignore hunk metadata lines
+                if (line.startsWith('\\')) continue;
+                // Context lines on the right side are also valid targets
+                validLines.add(`${file.filename}:${currentLine}`);
+                currentLine++;
               }
             }
 
-            // Partition comments into valid (on diff lines) and overflow
+            // Partition comments into valid (on right-side diff lines) and dropped
             const comments = Array.isArray(review.comments) ? review.comments : [];
             const validComments = [];
-            const overflowComments = [];
+            const droppedComments = [];
 
             for (const comment of comments) {
               const key = `${comment.path}:${comment.line}`;
@@ -114,19 +121,13 @@ jobs:
                   side: 'RIGHT',
                 });
               } else {
-                overflowComments.push(comment);
+                droppedComments.push(comment);
               }
             }
 
-            // Build review body: summary + any overflow comments
+            // Build review body from summary only.
+            // Intentionally do NOT publish out-of-diff comments.
             let body = review.summary || 'Codex review complete.';
-            if (overflowComments.length > 0) {
-              body += '\n\n### Additional comments\n';
-              body += '_These comments target lines outside the diff and could not be posted inline._\n\n';
-              for (const c of overflowComments) {
-                body += `- **\`${c.path}:${c.line}\`** — ${c.body}\n`;
-              }
-            }
 
             // Post the review
             await github.rest.pulls.createReview({
@@ -138,4 +139,4 @@ jobs:
               comments: validComments,
             });
 
-            console.log(`Review posted: ${validComments.length} inline comments, ${overflowComments.length} overflow comments`);
+            console.log(`Review posted: ${validComments.length} inline comments, ${droppedComments.length} dropped out-of-diff comments`);

apps/agent/package.json

@@ -55,7 +55,6 @@
     "better-sqlite3": "^12.2.0",
     "dkg.js": "^8.2.2",
     "dotenv": "^16.3.1",
-    "mysql2": "^3.6.5",
     "drizzle-orm": "^0.44.7",
     "expo": "53.0.20",
     "expo-blur": "~14.1.5",
@@ -77,6 +76,7 @@
     "expo-three": "^8.0.0",
     "expo-web-browser": "~14.2.0",
     "js-sha256": "^0.11.1",
+    "mysql2": "^3.6.5",
     "nodemailer": "^7.0.6",
     "prompts": "^2.4.2",
     "react": "19.0.0",

apps/agent/src/server/index.ts

@@ -32,7 +32,14 @@ const version = "1.0.0";
 const { oauthPlugin, openapiSecurityScheme } = createOAuthPlugin({
   storage: new SqliteOAuthStorageProvider(db),
   issuerUrl: new URL(process.env.EXPO_PUBLIC_MCP_URL),
-  scopesSupported: ["mcp", "llm", "scope123", "blob"],
+  scopesSupported: [
+    "mcp",
+    "llm",
+    "scope123",
+    "blob",
+    "epcis.read",
+    "epcis.write",
+  ],
   loginPageUrl: new URL(process.env.EXPO_PUBLIC_APP_URL + "/login"),
   schema: userCredentialsSchema,
   async login(credentials) {
@@ -101,6 +108,12 @@ const app = createPluginServer({
     oauthPlugin,
     (_, __, api) => {
       api.use("/mcp", authorized(["mcp"]));
+      api.use("/mcp", (req, res, next) => {
+        if (res.locals.auth) {
+          (req as any).auth = res.locals.auth;
+        }
+        next();
+      });
       api.use("/llm", authorized(["llm"]));
       api.use("/blob", authorized(["blob"]));
       api.use("/change-password", authorized([]));

apps/agent/src/server/scripts/setup.ts

@@ -9,8 +9,8 @@ import {
 import {
   getLLMProviderApiKeyEnvName,
   LLMProvider,
-  DEFAULT_SYSTEM_PROMPT,
 } from "@/shared/chat";
+import { DEFAULT_SYSTEM_PROMPT } from "@/shared/prompts/defaultSystemPrompt";
 
 async function setup() {
   const r = await prompts([
@@ -50,6 +50,25 @@ async function setup() {
       initial: DEFAULT_SYSTEM_PROMPT,
       format: (val) => (val === DEFAULT_SYSTEM_PROMPT ? "" : val.trim()),
     },
+    {
+      type: "select",
+      name: "docConversionProvider",
+      message: "Document conversion provider",
+      choices: [
+        { title: "unpdf — basic PDF only", value: "unpdf" },
+        { title: "Mistral OCR — complex PDF/DOCX/PPTX", value: "mistral" },
+      ],
+      initial: 0,
+    },
+    {
+      type: (_, a) =>
+        a.docConversionProvider === "mistral" && a.llmProvider !== "mistralai"
+          ? "text"
+          : null,
+      name: "mistralApiKey",
+      message: "MISTRAL_API_KEY",
+      validate: (val) => val.length || "Required for Mistral OCR provider",
+    },
     {
       type: "select",
       name: "dkgEnv",
@@ -132,8 +151,9 @@ async function setup() {
     {
       type: "text",
       name: "dbFilename",
-      message: "Database filename (i.e: example.db)",
+      message: "Database filename (e.g. example.db)",
       validate: (val) => val.length || "Required",
+      format: (val) => (val.endsWith(".db") ? val : `${val}.db`),
     },
   ]);
 
@@ -158,7 +178,8 @@ SMTP_USER="${r.smtpUsername || ""}"
 SMTP_PASS="${r.smtpPassword || ""}"
 SMTP_SECURE=${r.smtpSecure === undefined ? "true" : r.smtpSecure}
 SMTP_FROM="${r.smtpFrom || ""}"
-`,
+DOCUMENT_CONVERSION_PROVIDER="${r.docConversionProvider}"
+${r.docConversionProvider === "mistral" && r.llmProvider !== "mistralai" ? `MISTRAL_API_KEY="${r.mistralApiKey}"\n` : ""}`,
   );
 
   console.log("Creating .env.development.local file...");

apps/agent/src/shared/chat.ts

@@ -9,6 +9,8 @@ import type {
 import type { ToolCallChunk } from "@langchain/core/messages/tool";
 import type { CallToolResult } from "@modelcontextprotocol/sdk/types.js";
 
+import { DEFAULT_SYSTEM_PROMPT } from "./prompts/defaultSystemPrompt";
+
 export type { ToolDefinition };
 export type ToolInfo = {
   name: string;
@@ -339,7 +341,7 @@ export const processStreamingCompletion = async (
           try {
             args = tc.args ? JSON.parse(tc.args) : {};
           } catch {
-            // Malformed JSON from partial streaming — send raw
+            // Malformed JSON from partial streaming - send raw
             args = {};
           }
           toolCalls.push({
@@ -358,17 +360,17 @@ export const processStreamingCompletion = async (
       writeSSE(res, { event: "done", data: {} });
     } catch (streamError) {
       if (hasSentContent) {
-        // Partial content was already sent — don't re-invoke and risk
+        // Partial content was already sent - don't re-invoke and risk
         // duplicated/mixed output. Send an error so the UI can recover.
         writeSSE(res, {
           event: "error",
           data: {
             message:
-              "Stream interrupted — please retry your message",
+              "Stream interrupted - please retry your message",
           },
         });
       } else {
-        // No content sent yet — safe to fallback to a full invoke
+        // No content sent yet - safe to fallback to a full invoke
         try {
           const result = await provider.invoke(messages, options);
           const content = result.content;
@@ -514,42 +516,9 @@ export const makeStreamingCompletionRequest = async (
 
     // Stream ended without an explicit done/error event (server crash, network drop)
     if (!streamFinalized) {
-      callbacks.onError("Connection lost — the server stopped responding");
+      callbacks.onError("Connection lost - the server stopped responding");
     }
   } finally {
     reader.releaseLock();
   }
 };
-
-export const DEFAULT_SYSTEM_PROMPT = `
-You are a DKG Agent that helps users interact with the OriginTrail Decentralized Knowledge Graph (DKG) using available Model Context Protocol (MCP) tools.
-Your role is to help users create, retrieve, and analyze verifiable knowledge in a friendly, approachable, and knowledgeable way, making the technology accessible to both experts and non-experts. When replying, use markdown (e.g. bold text, bullet points, tables, etc.) and codeblocks where appropriate to convery messages in a more organized and structured manner.
-
-## Core Responsibilities
-- Answer Questions: Retrieve and explain knowledge from the DKG to help users understand and solve problems.
-- Create Knowledge Assets: Assist users in publishing new knowledge assets to the DKG using MCP tools.
-- Perform Analyses: Use DKG data and MCP tools to perform structured analyses, presenting results clearly.
-- Be Helpful and Approachable: Communicate in simple, user-friendly terms. Use analogies and clear explanations where needed, but avoid unnecessary technical jargon unless requested.
-
-## Privacy Rule (IMPORTANT)
-When creating or publishing knowledge assets:
-- If privacy is explicitly specified, follow the user’s instruction.
-- If privacy is NOT specified, ALWAYS set privacy to "private".
-- NEVER default to "public" without explicit user consent.
-This ensures sensitive information is not unintentionally exposed.
-
-## Interaction Guidelines
-1. Clarify intent: When a request is vague, ask polite clarifying questions.
-2. Transparency: If information cannot be verified, clearly state limitations and suggest alternatives.
-3. Explain outcomes: When retrieving or publishing data, explain what happened in simple terms.
-4. Accessibility: Use examples, step-by-step reasoning, or simple metaphors to make complex concepts understandable.
-5. Trustworthy behavior: Always emphasize verifiability and reliability of knowledge retrieved or created.
-
-## Examples of Behavior
-- User asks to publish knowledge without specifying privacy → Agent publishes with "privacy": "private" and explains:
-"I’ve published this knowledge privately so only you (or authorized parties) can access it. If you’d like it public, just let me know."
-
-- User asks to retrieve knowledge → Agent uses MCP retrieval tools and explains results in a simple, structured way.
-
-- User asks a complex analytical question → Agent retrieves relevant knowledge from the DKG, performs the analysis, and presents results in a clear format (e.g., list, table, etc.).
-`.trim();