Skip to content

ENG-1557: Optimize getAllDiscourseNodesSince to single query #910

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sid597 wants to merge 2 commits into
base: main
Choose a base branch
from
+74 −59
Open

ENG-1557: Optimize getAllDiscourseNodesSince to single query #910

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ee083fd
ENG-1557: Optimize getAllDiscourseNodesSince to single query
sid597 Mar 23, 2026
be3b89c
Fix block-backed discourse node sync
sid597 Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 74 additions & 59 deletions apps/roam/src/utils/getAllDiscourseNodesSince.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { type DiscourseNode } from "./getDiscourseNodes";
import getDiscourseNodeFormatExpression from "./getDiscourseNodeFormatExpression";
import { extractRef } from "roamjs-components/util";
import extractRef from "roamjs-components/util/extractRef";

type ISODateString = string;

Expand All @@ -17,22 +17,21 @@ export type RoamDiscourseNodeData = {
};
/* eslint-enable @typescript-eslint/naming-convention */

export type DiscourseNodesSinceResult = {
pageNodes: RoamDiscourseNodeData[];
blockNodes: RoamDiscourseNodeData[];
};

export const getDiscourseNodeTypeWithSettingsBlockNodes = async (
node: DiscourseNode,
sinceMs: number,
): Promise<RoamDiscourseNodeData[]> => {
const firstChildUid = extractRef(node.embeddingRef || "");
if (!firstChildUid) {
return [];
}

const regex = getDiscourseNodeFormatExpression(node.format);
const regexPattern = regex.source.replace(/\\/g, "\\\\").replace(/"/g, '\\"');
const firstChildUid = extractRef(node.embeddingRef);
const queryBlock = `[
:find ?childString ?nodeUid ?nodeCreateTime ?nodeEditTime ?author_local_id ?type ?author_name ?node-title
:keys text source_local_id created last_modified author_local_id type author_name node_title
:in $ ?firstChildUid ?type ?since
:find ?childString ?nodeUid ?nodeCreateTime ?nodeEditTime ?author_local_id ?author_name ?node-title
:keys text source_local_id created last_modified author_local_id author_name node_title
:in $ ?firstChildUid ?since
:where
[(re-pattern "${regexPattern}") ?title-regex]
[?node :node/title ?node-title]
Expand All @@ -57,75 +56,91 @@ export const getDiscourseNodeTypeWithSettingsBlockNodes = async (
[(> ?nodeEditTime ?since)]]
]`;

const blockNode = (await window.roamAlphaAPI.data.backend.q(
return (await window.roamAlphaAPI.data.backend.q(
queryBlock,
String(firstChildUid),
String(node.type),
sinceMs,
)) as unknown[] as RoamDiscourseNodeData[];
return blockNode;
};

export const getAllDiscourseNodesSince = async (
since: ISODateString,
nodeTypes: DiscourseNode[],
): Promise<RoamDiscourseNodeData[]> => {
const sinceMs = new Date(since).getTime();
if (!nodeTypes.length) {
return [];
}

const typeMatchers = nodeTypes.map((node) => ({
node,
regex: getDiscourseNodeFormatExpression(node.format),
}));
const regexPattern = typeMatchers
.map(({ regex }) => `(?:${regex.source})`)
.join("|")
.replace(/\\/g, "\\\\")
.replace(/"/g, '\\"');

const query = `[
:find ?node-title ?uid ?nodeCreateTime ?nodeEditTime ?author_local_id ?author_name
:keys text source_local_id created last_modified author_local_id author_name
:in $ ?since
:where
[(re-pattern "${regexPattern}") ?title-regex]
[?node :node/title ?node-title]
[(re-find ?title-regex ?node-title)]
[?node :block/uid ?uid]
[?node :create/time ?nodeCreateTime]
[?node :create/user ?user-eid]
[?user-eid :user/uid ?author_local_id]
[(get-else $ ?user-eid :user/display-name "Anonymous User") ?author_name]
[(get-else $ ?node :edit/time ?nodeCreateTime) ?nodeEditTime]
[(> ?nodeEditTime ?since)]
]`;

const allPages = (await window.roamAlphaAPI.data.backend.q(
query,
sinceMs,
)) as unknown[] as RoamDiscourseNodeData[];

const resultMap = new Map<string, RoamDiscourseNodeData>();
const blockBackedNodeTypes = nodeTypes.filter((node) =>
Boolean(extractRef(node.embeddingRef || "")),
);

for (const page of allPages) {
for (const { node, regex } of typeMatchers) {
if (regex.test(page.text)) {
if (page.source_local_id) {
resultMap.set(page.source_local_id, {
...page,
type: node.type,
});
}
break;
}
}
}

await Promise.all(
Copy link
Collaborator Author

@sid597 sid597 Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I spent time on this as well but it did not work out. The block lookup is more constrained, we need to know which specific page uid matched to the node type and which specific embeddingRef setting belongs to that node type.

I tried all at once query:

  • passing paired tupple inputs into the query .
  • passing all the page uids and all embedding-ref uids into one query and then filtering

both did not work return 0 for the nodes that had embedding ref enabled.

We can figure this out later when we get to this in future, a user reports etc. or create a new ticket spevifically for this

nodeTypes.map(async (node) => {
const regex = getDiscourseNodeFormatExpression(node.format);
const regexPattern = regex.source
.replace(/\\/g, "\\\\")
.replace(/"/g, '\\"');

const query = `[
:find ?node-title ?uid ?nodeCreateTime ?nodeEditTime ?author_local_id ?author_name ?type
:keys text source_local_id created last_modified author_local_id author_name type
:in $ ?since ?type
:where
[(re-pattern "${regexPattern}") ?title-regex]
[?node :node/title ?node-title]
[(re-find ?title-regex ?node-title)]
[?node :block/uid ?uid]
[?node :create/time ?nodeCreateTime]
[?node :create/user ?user-eid]
[?user-eid :user/uid ?author_local_id]
[(get-else $ ?user-eid :user/display-name "Anonymous User") ?author_name]
[(get-else $ ?node :edit/time ?nodeCreateTime) ?nodeEditTime]
[(> ?nodeEditTime ?since)]
]`;

const nodesOfType = (await window.roamAlphaAPI.data.backend.q(
query,
blockBackedNodeTypes.map(async (node) => {
const blockNodes = await getDiscourseNodeTypeWithSettingsBlockNodes(
node,
sinceMs,
String(node.type),
)) as unknown[] as RoamDiscourseNodeData[];

nodesOfType.forEach((n) => {
if (n.source_local_id) {
resultMap.set(n.source_local_id, n);
}
});
);

const hasBlockSettings =
node.embeddingRef && extractRef(node.embeddingRef);
if (hasBlockSettings) {
const blockNodes = await getDiscourseNodeTypeWithSettingsBlockNodes(
node,
sinceMs,
);
if (blockNodes) {
blockNodes.forEach((blockNode) => {
if (blockNode.source_local_id) {
resultMap.set(blockNode.source_local_id, blockNode);
}
blockNodes.forEach((blockNode) => {
if (blockNode.source_local_id) {
resultMap.set(blockNode.source_local_id, {
...blockNode,
type: node.type,
});
}
}
});
}),
);

return Array.from(resultMap.values());
};

Expand Down
Loading