fix(persistence): chunk block and edge inserts to prevent sql variabl…

[Siddhartha-singh01]

Summary

Fixes the bug where workflows disappear or show up empty after a page refresh in self-hosted mode (SQLite).

The issue was that saveWorkflowToNormalizedTables was trying to bulk insert all blocks at once. SQLite has a hard limit of 999 parameters per query. Since every block has 17 fields, saving workflows with around 60 blocks was hitting the limit and silently failing the save transaction.

I just added a quick fix to chunk the array into smaller batches of 50 before inserting them.

Fixes #2424

[cursor]

PR Summary

Low Risk
Low risk change that only alters how inserts are batched during workflow saves; behavior should remain identical but reduces SQLite query parameter limit failures.

Overview
Prevents workflow saves from failing on SQLite by batching inserts into workflowBlocks, workflowEdges, and workflowSubflows within saveWorkflowToNormalizedTables.

Instead of single bulk inserts, the transaction now writes records in fixed-size chunks (CHUNK_SIZE = 50), avoiding oversized insert statements while keeping the same delete-and-reinsert flow.

^{Written by Cursor Bugbot for commit 160dc1c. This will update automatically on new commits. Configure here.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Mar 5, 2026 10:51pm

[greptile-apps]

Greptile Summary

This PR fixes a critical data-loss bug in self-hosted (SQLite) deployments where workflows would disappear after a page refresh. The root cause was that saveWorkflowToNormalizedTables attempted to bulk-insert all blocks, edges, and subflows in a single SQL statement, hitting SQLite's hard limit of 999 bound parameters and causing the transaction to fail silently.

The fix introduces a CHUNK_SIZE = 50 constant and replaces the single bulk inserts with sequential chunked inserts inside the existing db.transaction, preserving full atomicity (any chunk failure rolls back the entire transaction).

Key points:

• With 17 fields per block, a chunk of 50 yields 850 parameters — safely under the 999-parameter ceiling.
• Edges (6 fields) and subflows (4 fields) are also chunked at 50, well under the limit.
• Chunking is applied globally (PostgreSQL and SQLite alike), introducing minimal overhead for non-SQLite deployments.
• The implementation is logically correct and transaction atomicity is fully preserved.

Style feedback:

• CHUNK_SIZE is currently defined inside the db.transaction callback; moving it to module scope would be cleaner and make the intent clearer.
• The derivation of chunk size should be documented with a comment, since if new fields are added to workflowBlocks without updating CHUNK_SIZE, the code will silently fail.

Confidence Score: 4/5

• The fix is logically correct and safe to merge; it properly handles SQLite's parameter limit constraints while preserving transaction atomicity.
• The core fix is solid—chunking block, edge, and subflow inserts prevents hitting SQLite's 999-parameter limit and fully preserves transaction atomicity. The only remaining feedback is stylistic: moving CHUNK_SIZE to module scope and documenting the chunk size derivation. These are minor improvements that don't affect correctness or safety.
• No files require special attention. The changes are isolated to saveWorkflowToNormalizedTables and are minimal in scope.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[saveWorkflowToNormalizedTables called] --> B[Begin db.transaction]
    B --> C[DELETE existing blocks / edges / subflows]
    C --> D{blocks.length > 0?}
    D -- Yes --> E[Build blockInserts array]
    E --> F[Loop: slice chunks of 50\ninsert each chunk into workflowBlocks]
    D -- No --> G{edges.length > 0?}
    F --> G
    G -- Yes --> H[Build edgeInserts array]
    H --> I[Loop: slice chunks of 50\ninsert each chunk into workflowEdges]
    G -- No --> J[Build subflowInserts array]
    I --> J
    J --> K{subflowInserts.length > 0?}
    K -- Yes --> L[Loop: slice chunks of 50\ninsert each chunk into workflowSubflows]
    K -- No --> M[Commit transaction]
    L --> M
    M --> N[Return success: true]

    style F fill:#d4edda,stroke:#28a745
    style I fill:#d4edda,stroke:#28a745
    style L fill:#d4edda,stroke:#28a745

Loading

_{Last reviewed commit: 160dc1c}

[greptile-apps]

CHUNK_SIZE is defined inside the db.transaction async callback, which means it's re-created on every call to saveWorkflowToNormalizedTables. Since it's a fixed, never-changing value, consider hoisting it to module scope. This makes the intent clearer and avoids unnecessary re-allocation.

Suggested change

	const CHUNK_SIZE = 50
	const CHUNK_SIZE = 50

Place this near the top of the file alongside other module-level constants (e.g., after the logger definition on line 22).

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

[greptile-apps]

The SQLite hard limit of 999 bound parameters per statement is implicit in the choice of CHUNK_SIZE = 50, but this constraint is not documented. With 17 fields per block record, a chunk of 50 yields 850 parameters—safely under the limit. However, if new fields are ever added to workflowBlocks without updating CHUNK_SIZE, the code will silently fail. Consider documenting the derivation:

Suggested change

	for (let i = 0; i < blockInserts.length; i += CHUNK_SIZE) {
	await tx.insert(workflowBlocks).values(blockInserts.slice(i, i + CHUNK_SIZE))
	}
	// SQLite limits bound parameters to 999 per statement.
	// workflowBlocks has 17 fields → max safe chunk = floor(999/17) = 58.
	// Using 50 for a conservative margin.
	for (let i = 0; i < blockInserts.length; i += CHUNK_SIZE) {
	await tx.insert(workflowBlocks).values(blockInserts.slice(i, i + CHUNK_SIZE))
	}

[icecrasher321]

@Siddhartha-singh01 please point this to the staging branch not main