Skip to content

feat: full-text search with FlexSearch #102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
s-adamantine wants to merge 9 commits into
base: main
Choose a base branch
from
Open

feat: full-text search with FlexSearch #102

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
6417d68
Add build-time search index generator (hypercerts-atproto-documentati…
s-adamantine Mar 9, 2026
5cff476
beads: close hypercerts-atproto-documentation-7fu.1
s-adamantine Mar 9, 2026
7cb8d0e
Replace SearchDialog with FlexSearch-powered full-text search (hyperc…
s-adamantine Mar 9, 2026
4fcd51b
beads: close hypercerts-atproto-documentation-7fu.2
s-adamantine Mar 9, 2026
9f49190
Add CSS for search result snippets and mark highlighting (hypercerts-…
s-adamantine Mar 9, 2026
882a76e
beads: close hypercerts-atproto-documentation-7fu.3
s-adamantine Mar 9, 2026
f1b54b2
Update lastUpdated.json from build
s-adamantine Mar 9, 2026
1ed7c96
fix: join headings array to string before FlexSearch indexing
s-adamantine Mar 9, 2026
5b90aa9
fix: FlexSearch results are plain IDs not objects, handle both formats
s-adamantine Mar 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .beads/issues.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

183 changes: 154 additions & 29 deletions components/SearchDialog.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,24 @@
import { useState, useEffect, useRef, useCallback } from 'react';
import { useRouter } from 'next/router';
import { flattenNavigation } from '../lib/navigation';

// Fuzzy match: all characters of query appear in text in order (case-insensitive)
function fuzzyMatch(query, text) {
const q = query.toLowerCase();
const t = text.toLowerCase();
let qi = 0;
for (let ti = 0; ti < t.length && qi < q.length; ti++) {
if (t[ti] === q[qi]) qi++;
}
return qi === q.length;
import FlexSearch from 'flexsearch';

// Module-level variables to persist across dialog opens
let searchData = null;
let searchIndex = null;
let isLoading = false;

// Generate a context snippet with the matched term highlighted
function getSnippet(body, query, contextChars = 60) {
const lower = body.toLowerCase();
const idx = lower.indexOf(query.toLowerCase());
if (idx === -1) return '';
const start = Math.max(0, idx - contextChars);
const end = Math.min(body.length, idx + query.length + contextChars);
let snippet = '';
if (start > 0) snippet += '...';
snippet += body.slice(start, end);
if (end < body.length) snippet += '...';
return snippet;
}

// Paths for the curated quick links shown in the empty state
Expand All @@ -26,31 +34,119 @@ const QUICK_LINK_PATHS = [
export function SearchDialog({ isOpen, onClose }) {
const [query, setQuery] = useState('');
const [selectedIndex, setSelectedIndex] = useState(-1);
const [loading, setLoading] = useState(false);
const [results, setResults] = useState([]);
const inputRef = useRef(null);
const itemRefs = useRef([]);
const router = useRouter();
const allPages = flattenNavigation();
const debounceTimerRef = useRef(null);

// Quick links for empty state
const quickLinks = QUICK_LINK_PATHS
.map(p => allPages.find(page => page.path === p))
.filter(Boolean);

// Filter and sort results using fuzzy matching
const trimmedQuery = query.trim();
const results = trimmedQuery.length > 0
? allPages
.filter(p => fuzzyMatch(trimmedQuery, p.title))
.sort((a, b) => {
const aExact = a.title.toLowerCase().includes(trimmedQuery.toLowerCase());
const bExact = b.title.toLowerCase().includes(trimmedQuery.toLowerCase());
if (aExact && !bExact) return -1;
if (!aExact && bExact) return 1;
return 0;
// Load search index on first open
useEffect(() => {
if (isOpen && !searchData && !isLoading) {
isLoading = true;
setLoading(true);
fetch('/search-index.json')
.then(res => res.json())
.then(data => {
searchData = data;
// Create FlexSearch index
searchIndex = new FlexSearch.Document({
document: {
id: 'path',
index: ['title', 'description', 'headings', 'body'],
},
tokenize: 'forward',
});
// Add all entries to the index — join headings array into string for FlexSearch
data.forEach(entry => {
searchIndex.add({
...entry,
headings: Array.isArray(entry.headings) ? entry.headings.join(' ') : (entry.headings || ''),
});
});
setLoading(false);
isLoading = false;
})
.catch(err => {
console.error('Failed to load search index:', err);
setLoading(false);
isLoading = false;
});
Comment on lines +71 to +75
Copy link

@coderabbitai coderabbitai bot Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Silent failure when index fails to load.

If the fetch fails, the error is only logged to the console. Users see an empty quick links state (since searchData remains null, quickLinks will be empty) with no indication that something went wrong.

🛡️ Proposed fix to show error state 
+  const [error, setError] = useState(null);
+
   // Load search index on first open
   useEffect(() => {
     if (isOpen && !searchData && !isLoading) {
       isLoading = true;
       setLoading(true);
+      setError(null);
       fetch('/search-index.json')
         .then(res => res.json())
         // ... existing success handling ...
         .catch(err => {
           console.error('Failed to load search index:', err);
+          setError('Failed to load search index');
           setLoading(false);
           isLoading = false;
         });
     }
   }, [isOpen]);

Then in the render:

 {loading ? (
   <div className="search-no-results">Loading...</div>
+) : error ? (
+  <div className="search-no-results">{error}</div>
 ) : !hasQuery ? (
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@components/SearchDialog.js` around lines 71 - 75, The current catch in the
search index load only logs the error and leaves searchData null, causing an
empty quickLinks UI without feedback; update the catch to set a visible error
state (e.g., setSearchError or setError), flip setLoading(false) and isLoading =
false, and set searchData to an empty array or a sentinel so quickLinks
rendering can show a user-facing error message; update the render logic that
reads searchData / quickLinks to display the new error state (use the new
setSearchError flag) instead of silently showing empty quick links.

}
}, [isOpen]);

// Quick links for empty state
const quickLinks = searchData
? QUICK_LINK_PATHS
.map(p => searchData.find(page => page.path === p))
.filter(Boolean)
: [];

// Perform search with debouncing
useEffect(() => {
const trimmedQuery = query.trim();

if (debounceTimerRef.current) {
clearTimeout(debounceTimerRef.current);
}

if (!trimmedQuery || !searchIndex || !searchData) {
setResults([]);
return;
}

debounceTimerRef.current = setTimeout(() => {
// Search the index
const searchResults = searchIndex.search(trimmedQuery, { limit: 20, enrich: true });

// Merge and deduplicate results by path, prioritizing field order
const pathMap = new Map();
const fieldPriority = { title: 1, description: 2, headings: 3, body: 4 };

searchResults.forEach(fieldResult => {
const field = fieldResult.field;
const priority = fieldPriority[field] || 999;

fieldResult.result.forEach(item => {
const path = typeof item === 'object' ? item.id : item;
if (!pathMap.has(path) || pathMap.get(path).priority > priority) {
const entry = searchData.find(e => e.path === path);
if (entry) {
pathMap.set(path, {
...entry,
priority,
matchedField: field,
});
}
}
});
});

// Convert to array and sort by priority
const mergedResults = Array.from(pathMap.values()).sort((a, b) => a.priority - b.priority);

// Generate snippets for body matches
mergedResults.forEach(result => {
if (result.matchedField === 'body' && result.body) {
result.snippet = getSnippet(result.body, trimmedQuery);
} else if (result.description) {
result.snippet = result.description;
}
});

setResults(mergedResults);
}, 150);

return () => {
if (debounceTimerRef.current) {
clearTimeout(debounceTimerRef.current);
}
};
}, [query]);

// Group results by section
const trimmedQuery = query.trim();
const groupedResults = results.reduce((acc, page) => {
const section = page.section || 'General';
if (!acc[section]) acc[section] = [];
Expand Down Expand Up @@ -107,6 +203,27 @@ export function SearchDialog({ isOpen, onClose }) {
// Build a flat index counter for refs across groups
let refIndex = 0;

// Helper to highlight matched term in snippet
const highlightSnippet = (snippet, query) => {
if (!snippet || !query) return snippet;
const lower = snippet.toLowerCase();
const queryLower = query.toLowerCase();
const idx = lower.indexOf(queryLower);
if (idx === -1) return snippet;

const before = snippet.slice(0, idx);
const match = snippet.slice(idx, idx + query.length);
const after = snippet.slice(idx + query.length);

return (
<>
{before}
<mark>{match}</mark>
{after}
</>
);
};

return (
<div className="search-overlay" onClick={onClose}>
<div className="search-dialog" onClick={(e) => e.stopPropagation()}>
Expand Down Expand Up @@ -137,7 +254,10 @@ export function SearchDialog({ isOpen, onClose }) {
<kbd className="search-input-kbd">ESC</kbd>
</div>
<div className="search-results">
{!hasQuery ? (
{loading ? (
/* Loading state */
<div className="search-no-results">Loading...</div>
) : !hasQuery ? (
/* Empty state: show Quick Links */
<ul className="search-results-list">
<li>
Expand Down Expand Up @@ -178,6 +298,11 @@ export function SearchDialog({ isOpen, onClose }) {
type="button"
>
<span className="search-result-title">{page.title}</span>
{page.snippet && (
<span className="search-result-snippet">
{highlightSnippet(page.snippet, trimmedQuery)}
</span>
)}
<span className="search-result-path">{page.path}</span>
</button>
</li>
Expand Down
139 changes: 139 additions & 0 deletions lib/generate-search-index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
const { readdirSync, statSync, readFileSync, writeFileSync } = require("fs");
const { join, relative } = require("path");

const PAGES_DIR = join(__dirname, "..", "pages");
const OUTPUT = join(__dirname, "..", "public", "search-index.json");
const MAX_BODY_LENGTH = 5000;

function walkDir(dir) {
const results = [];
for (const entry of readdirSync(dir)) {
const full = join(dir, entry);
if (statSync(full).isDirectory()) {
results.push(...walkDir(full));
} else if (full.endsWith(".md")) {
results.push(full);
}
}
return results;
}

function extractFrontmatter(content) {
const fmMatch = content.match(/^---\n([\s\S]*?)\n---/);
if (!fmMatch) return { title: "", description: "" };

const frontmatter = fmMatch[1];
const titleMatch = frontmatter.match(/^title:\s*(.+)$/m);
const descMatch = frontmatter.match(/^description:\s*(.+)$/m);

return {
title: titleMatch ? titleMatch[1].trim() : "",
description: descMatch ? descMatch[1].trim() : "",
};
}

function extractHeadings(content) {
const headings = [];
const lines = content.split("\n");

for (const line of lines) {
// Match h2 (## ) or h3 (### )
const h2Match = line.match(/^##\s+(.+)$/);
const h3Match = line.match(/^###\s+(.+)$/);

if (h2Match) {
headings.push(h2Match[1].trim());
} else if (h3Match) {
headings.push(h3Match[1].trim());
}
}

return headings;
}

function stripMarkdown(content) {
let text = content;

// Remove frontmatter
text = text.replace(/^---\n[\s\S]*?\n---\n?/, "");

// Remove code blocks
text = text.replace(/```[\s\S]*?```/g, "");

// Remove Markdoc tags ({% ... %} and {% /... %})
text = text.replace(/\{%[\s\S]*?%\}/g, "");

// Remove HTML tags
text = text.replace(/<[^>]+>/g, "");

// Remove heading markers
text = text.replace(/^#{1,6}\s+/gm, "");

// Remove markdown links [text](url) -> text
text = text.replace(/\[([^\]]+)\]\([^\)]+\)/g, "$1");

// Remove bold/italic markers
text = text.replace(/\*\*([^*]+)\*\*/g, "$1");
text = text.replace(/\*([^*]+)\*/g, "$1");
text = text.replace(/__([^_]+)__/g, "$1");
text = text.replace(/_([^_]+)_/g, "$1");

// Remove inline code backticks
text = text.replace(/`([^`]+)`/g, "$1");

// Collapse whitespace
text = text.replace(/\s+/g, " ");

return text.trim();
}

function getSection(path) {
if (path === "/") return "Get Started";
if (path.startsWith("/getting-started")) return "Get Started";
if (path.startsWith("/core-concepts")) return "Core Concepts";
if (path.startsWith("/tools")) return "Tools";
if (path.startsWith("/architecture")) return "Architecture";
if (path.startsWith("/lexicons")) return "Reference";
if (path.startsWith("/reference")) return "Reference";
if (path.startsWith("/ecosystem")) return "Ecosystem & Vision";
if (path === "/roadmap") return "Reference";
return "Other";
}

const files = walkDir(PAGES_DIR);
const index = [];

for (const file of files) {
const content = readFileSync(file, "utf-8");
const rel = "/" + relative(PAGES_DIR, file).replace(/\.md$/, "");
const path = rel === "/index" ? "/" : rel;

const { title, description } = extractFrontmatter(content);
const headings = extractHeadings(content);
const section = getSection(path);

// For the home page, only include title (body is mostly card markup)
let body = "";
if (path !== "/") {
body = stripMarkdown(content);
if (body.length > MAX_BODY_LENGTH) {
body = body.substring(0, MAX_BODY_LENGTH);
}
}

index.push({
path,
title,
description: description || "",
section,
headings,
body,
});
}

writeFileSync(OUTPUT, JSON.stringify(index, null, 2) + "\n");
console.log(
`Generated search index for ${index.length} pages (${
Buffer.byteLength(JSON.stringify(index)) / 1024
} KB)`
);
4 changes: 2 additions & 2 deletions lib/lastUpdated.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
"/lexicons/hypercerts-lexicons/activity-claim": "2026-03-05T19:47:55+08:00",
"/lexicons/hypercerts-lexicons/attachment": "2026-03-05T19:50:36+08:00",
"/lexicons/hypercerts-lexicons/collection": "2026-03-05T20:17:36+06:00",
"/lexicons/hypercerts-lexicons/contribution": "2026-03-09T15:46:09+08:00",
"/lexicons/hypercerts-lexicons/contribution": "2026-03-09T15:56:42+08:00",
"/lexicons/hypercerts-lexicons/evaluation": "2026-03-05T19:51:39+08:00",
"/lexicons/hypercerts-lexicons/funding-receipt": "2026-03-05T19:47:55+08:00",
"/lexicons/hypercerts-lexicons/index": "2026-03-05T20:17:36+06:00",
Expand All @@ -38,7 +38,7 @@
"/reference/faq": "2026-03-06T22:56:46+08:00",
"/reference/glossary": "2026-03-05T20:14:26+08:00",
"/roadmap": "2026-03-05T20:17:36+06:00",
"/tools/hyperboards": "2026-03-05T17:11:22+06:00",
"/tools/hyperboards": "2026-03-09T15:56:42+08:00",
"/tools/hypercerts-cli": "2026-03-05T12:30:00+06:00",
"/tools/hyperindex": "2026-02-20T19:42:03+08:00",
"/tools/scaffold": "2026-03-05T15:58:36+06:00"
Expand Down
Loading