Skip to content

Commit 0604ab0

Browse files
atomanticatomantic
authored
Release v0.3.21
Release v0.3.21: SQLite Storage Layer, Real-Time Events, and Canonical Identity System
2 parents efc6068 + ff26fb6 commit 0604ab0

141 files changed

Lines changed: 21268 additions & 3558 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.changelog/v0.1.x.md renamed to .changelog/v0.1.1.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Release v0.1.x - Initial Release
1+
# Release v0.1.0 - Initial Release
22

33
Released: 2026-01-20
44

@@ -23,7 +23,7 @@ Initial release of SparseTree - a genealogy toolkit for creating local databases
2323
- GEDCOM import/export support
2424
- Light/dark theme support
2525

26-
### Infrastructure (v0.1.1)
26+
### Infrastructure
2727
- Browser status polling replaced with SSE
2828
- CDP browser integration for indexer
2929
- Ancestry tree line improvements
@@ -39,4 +39,4 @@ pm2 start ecosystem.config.cjs
3939

4040
## 🔗 Full Changelog
4141

42-
**Full Diff**: https://github.com/atomantic/SparseTree/releases/tag/v0.1.x
42+
**Full Diff**: https://github.com/atomantic/SparseTree/releases/tag/v0.1.0

.changelog/v0.2.x.md renamed to .changelog/v0.2.10.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Release v0.2.x - Multi-Platform Linking & Tree Views
1+
# Release v0.2.10 - Multi-Platform Linking & Tree Views
22

33
Released: YYYY-MM-DD
44

@@ -96,4 +96,4 @@ pm2 start ecosystem.config.cjs
9696

9797
## 🔗 Full Changelog
9898

99-
**Full Diff**: https://github.com/atomantic/SparseTree/compare/v0.1.x...v0.2.x
99+
**Full Diff**: https://github.com/atomantic/SparseTree/compare/v0.1.x...v0.2.10

‎.changelog/v0.3.x.md‎

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
# Release v0.3.x - SQLite Storage Layer & Migration Framework
2+
3+
Released: YYYY-MM-DD
4+
5+
## Overview
6+
7+
Major architectural upgrade introducing SQLite as a high-performance index layer while maintaining JSON files as the source of truth. This release adds canonical ULID-based identities, full-text search via FTS5, recursive CTEs for path finding, and a comprehensive data migration framework.
8+
9+
## 🎉 New Features
10+
11+
### SQLite Storage Layer
12+
- Added SQLite database (`data/sparsetree.db`) as fast query index
13+
- FTS5 full-text search for person names, bios, and occupations
14+
- Recursive CTEs for efficient ancestor/descendant path finding
15+
- WAL mode for better read concurrency
16+
- Auto-enables when database exists with data, falls back to JSON otherwise
17+
18+
### Canonical Identity System
19+
- ULID-based canonical IDs (26-char, sortable, collision-resistant)
20+
- External identity mappings for multiple providers (FamilySearch, Ancestry, WikiTree, 23andMe)
21+
- Bidirectional ID lookup with in-memory LRU cache
22+
- Confidence scoring for identity assertions
23+
24+
### Content-Addressed Blob Storage
25+
- SHA-256 hash-based media storage (`data/blobs/{hash[:2]}/{hash}.{ext}`)
26+
- Automatic deduplication of duplicate photos
27+
- Media records linked to persons with primary photo support
28+
29+
### Data Migration Framework
30+
- Automatic schema and data migrations on update
31+
- Migration tracking in `data/.data-version` and SQLite `migration` table
32+
- Dry-run mode for previewing changes
33+
- Rollback support for reversible migrations
34+
- Commands: `npm run migrate`, `npm run migrate:status`, `npm run migrate:dry-run`
35+
36+
### Update Script
37+
- New `./update.sh` for one-command updates
38+
- Pulls latest code, installs deps, builds, migrates, restarts
39+
- Supports `--dry-run`, `--no-restart`, `--branch=NAME` options
40+
- Safe: checks for uncommitted changes before updating
41+
42+
### Simplified Genealogy Provider UI (v0.3.10)
43+
- Consolidated genealogy provider management to a single `/providers/genealogy` page
44+
- Removed over-engineered `/providers/scraper` and `/providers/genealogy/:id/edit` routes
45+
- Simplified `BrowserSettingsPage` to focus on CDP connection settings only
46+
- Added "Login with Google" SSO option for FamilySearch
47+
- Browser connection controls now available directly on the providers page
48+
- Provider credentials and auto-login moved to the consolidated page
49+
50+
### Fixed Database Refresh Hang (v0.3.11)
51+
- Fixed refresh button hanging the server on large trees (138k+ persons)
52+
- Changed recursive CTE query to use database_membership table when available
53+
- Added hard limit (50 generations, 500k persons) to prevent runaway queries
54+
- Converted refresh to SSE-based background task to avoid HTTP timeouts
55+
- Client now uses EventSource for non-blocking refresh with progress updates
56+
57+
### Documentation Restructure (v0.3.12)
58+
- Created `docs/` folder with modular documentation:
59+
- `architecture.md` - Data model, storage, identity system
60+
- `api.md` - API endpoint reference
61+
- `cli.md` - CLI command reference
62+
- `development.md` - Development setup guide
63+
- `providers.md` - Genealogy provider configuration
64+
- `roadmap.md` - Detailed phase documentation
65+
- Simplified `PLAN.md` to high-level summary with links
66+
- Streamlined `CLAUDE.md` with quick reference format
67+
- Simplified `README.md` with user-focused content
68+
- Added Phase 17 (Socket.IO real-time event system) to roadmap
69+
70+
### Project Structure Consolidation (v0.3.13)
71+
- Moved root `lib/` directory into `server/src/lib/`:
72+
- `lib/config.ts` - Application configuration with TypeScript types
73+
- `lib/graph/` - Path finding algorithms (shortest, longest, random)
74+
- `lib/familysearch/` - FamilySearch API client, fetcher, and transformer
75+
- `lib/sqlite-writer.ts` - SQLite dual-write logic during indexing
76+
- Moved root CLI scripts to `scripts/`:
77+
- `index.ts` - Main ancestry indexer
78+
- `find.ts` - Path finder
79+
- `print.ts` - Chronological print
80+
- `purge.ts` - Purge cached records
81+
- `prune.ts` - Prune orphan files
82+
- `rebuild.ts` - Rebuild databases
83+
- `migrate-favorites.ts` - Favorites migration
84+
- Added `server/src/utils/sleep.ts` and `randInt.ts` utilities
85+
- Updated indexer.service.ts to spawn `npx tsx scripts/index.ts`
86+
- Removed deprecated `tsv.js` (redundant with `/api/export/:dbId/tsv`)
87+
- All scripts now run via `npx tsx` for TypeScript support
88+
89+
### Socket.IO Real-Time Events (v0.3.14)
90+
- Implemented Socket.IO for bidirectional real-time communication
91+
- Added `server/src/services/socket.service.ts` with room-based event broadcasting
92+
- Added `client/src/services/socket.ts` singleton socket client
93+
- Added `client/src/hooks/useSocket.ts` with React hooks for socket events:
94+
- `useSocketConnection()` - Manage socket connection lifecycle
95+
- `useSocketEvent()` - Subscribe to specific socket events
96+
- `useDatabaseEvents()` - Database refresh notifications
97+
- `useBrowserEvents()` - Browser status updates
98+
- `useIndexerEvents()` - Indexing progress
99+
- Converted database refresh from SSE to Socket.IO events
100+
- Browser status broadcasts via Socket.IO in addition to SSE (backwards compatible)
101+
- Added in-memory LRU cache for SQL queries (`server/src/services/cache.service.ts`):
102+
- Configurable TTL and max size
103+
- Separate caches for queries, persons, and lists
104+
- Cache invalidation per database or person
105+
- Cache statistics for monitoring
106+
- Optimized `/api/favorites` endpoint:
107+
- Replaced N+1 queries with single JOIN query
108+
- Removed loading of entire database contents
109+
- Added index on `favorite.added_at` for faster sorting
110+
- Added `person_count` column to `database_info` table
111+
112+
## 🔧 Improvements
113+
114+
### Service Layer Updates
115+
- `database.service.ts` - Queries from SQLite with JSON fallback
116+
- `search.service.ts` - Uses FTS5 for text search, supports cross-database global search
117+
- `path.service.ts` - Recursive CTEs for shortest/longest/random paths
118+
- `favorites.service.ts` - SQLite storage with JSON backup
119+
- `augmentation.service.ts` - External identity registration
120+
121+
### Schema Design
122+
- Normalized tables: `person`, `external_identity`, `parent_edge`, `spouse_edge`
123+
- Extensible claims system for facts with provenance
124+
- Vital events with date parsing (supports BC notation)
125+
- Database membership for multi-tree support
126+
- Full schema in `server/src/db/schema.sql`
127+
128+
### Developer Experience
129+
- Updated CLAUDE.md with comprehensive migration documentation
130+
- Added npm scripts for migration management
131+
- Better error messages for ID resolution failures
132+
133+
## 📦 Installation
134+
135+
```bash
136+
git clone https://github.com/atomantic/SparseTree.git
137+
cd SparseTree
138+
npm run install:all
139+
npm run migrate
140+
pm2 start ecosystem.config.cjs
141+
```
142+
143+
## 🔄 Upgrading from v0.2.x
144+
145+
Run the update script to automatically migrate:
146+
147+
```bash
148+
./update.sh
149+
```
150+
151+
Or manually:
152+
153+
```bash
154+
git pull origin main
155+
npm install
156+
npm run build
157+
npm run migrate
158+
pm2 restart ecosystem.config.cjs
159+
```
160+
161+
## 🔗 Full Changelog
162+
163+
**Full Diff**: https://github.com/atomantic/SparseTree/compare/v0.2.11...v0.3.x

‎.github/workflows/test.yml‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
name: Tests
2+
3+
on:
4+
push:
5+
branches: [dev, main]
6+
pull_request:
7+
branches: [dev, main]
8+
9+
jobs:
10+
unit-tests:
11+
name: Unit Tests
12+
runs-on: ubuntu-latest
13+
14+
steps:
15+
- uses: actions/checkout@v4
16+
17+
- name: Setup Node.js
18+
uses: actions/setup-node@v4
19+
with:
20+
node-version: '20'
21+
cache: 'npm'
22+
23+
- name: Install dependencies
24+
run: npm ci
25+
26+
- name: Run unit tests
27+
run: npm run test:unit
28+
29+
- name: Upload coverage
30+
uses: codecov/codecov-action@v4
31+
if: always()
32+
with:
33+
files: ./coverage/lcov.info
34+
fail_ci_if_error: false
35+
36+
integration-tests:
37+
name: Integration Tests
38+
runs-on: ubuntu-latest
39+
40+
steps:
41+
- uses: actions/checkout@v4
42+
43+
- name: Setup Node.js
44+
uses: actions/setup-node@v4
45+
with:
46+
node-version: '20'
47+
cache: 'npm'
48+
49+
- name: Install dependencies
50+
run: npm ci
51+
52+
- name: Run integration tests
53+
run: npm run test:integration
54+
55+
scraper-tests:
56+
name: Scraper Tests
57+
runs-on: ubuntu-latest
58+
59+
steps:
60+
- uses: actions/checkout@v4
61+
62+
- name: Setup Node.js
63+
uses: actions/setup-node@v4
64+
with:
65+
node-version: '20'
66+
cache: 'npm'
67+
68+
- name: Install dependencies
69+
run: npm ci
70+
71+
- name: Install Playwright browsers
72+
run: npx playwright install chromium
73+
74+
- name: Run scraper tests
75+
run: npm run test:scraper
76+
77+
- name: Upload test results
78+
uses: actions/upload-artifact@v4
79+
if: always()
80+
with:
81+
name: scraper-test-results
82+
path: test-results/
83+
retention-days: 7
84+
85+
all-tests:
86+
name: All Tests Summary
87+
runs-on: ubuntu-latest
88+
needs: [unit-tests, integration-tests, scraper-tests]
89+
if: always()
90+
91+
steps:
92+
- name: Check test results
93+
run: |
94+
if [[ "${{ needs.unit-tests.result }}" != "success" ]] || \
95+
[[ "${{ needs.integration-tests.result }}" != "success" ]] || \
96+
[[ "${{ needs.scraper-tests.result }}" != "success" ]]; then
97+
echo "One or more test jobs failed"
98+
exit 1
99+
fi
100+
echo "All tests passed!"

‎.github/workflows/validate-mocks.yml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
name: Validate Mock Selectors
2+
3+
on:
4+
schedule:
5+
# Run weekly on Mondays at 9am UTC
6+
- cron: '0 9 * * 1'
7+
workflow_dispatch:
8+
9+
jobs:
10+
validate:
11+
name: Validate Provider Selectors
12+
runs-on: ubuntu-latest
13+
14+
steps:
15+
- uses: actions/checkout@v4
16+
17+
- name: Setup Node.js
18+
uses: actions/setup-node@v4
19+
with:
20+
node-version: '20'
21+
cache: 'npm'
22+
23+
- name: Install dependencies
24+
run: npm ci
25+
26+
- name: Install Playwright browsers
27+
run: npx playwright install chromium
28+
29+
- name: Validate mock selectors
30+
id: validate
31+
continue-on-error: true
32+
run: npm run validate:mocks
33+
34+
- name: Create issue on failure
35+
if: steps.validate.outcome == 'failure'
36+
uses: actions/github-script@v7
37+
with:
38+
script: |
39+
const title = 'Mock Selector Validation Failed';
40+
const body = `## Mock Selector Validation Failed
41+
42+
The weekly validation of mock provider selectors has detected issues.
43+
44+
This means one or more genealogy providers have updated their website structure,
45+
and our mock templates may need to be updated to match.
46+
47+
### Action Required
48+
49+
1. Review the [workflow run](${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId})
50+
2. Update the affected selectors in \`tests/__mocks__/providers/*/selectors.json\`
51+
3. Update the corresponding HTML templates in \`tests/__mocks__/providers/*/pages/\`
52+
4. Re-run the scraper tests to verify
53+
54+
### Affected Run
55+
56+
- Workflow: ${context.workflow}
57+
- Run ID: ${context.runId}
58+
- Date: ${new Date().toISOString()}
59+
60+
/label mock-validation`;
61+
62+
// Check if issue already exists
63+
const { data: issues } = await github.rest.issues.listForRepo({
64+
owner: context.repo.owner,
65+
repo: context.repo.repo,
66+
labels: 'mock-validation',
67+
state: 'open'
68+
});
69+
70+
if (issues.length === 0) {
71+
await github.rest.issues.create({
72+
owner: context.repo.owner,
73+
repo: context.repo.repo,
74+
title,
75+
body,
76+
labels: ['mock-validation', 'maintenance']
77+
});
78+
}

‎.gitignore‎

Lines changed: 2 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,5 @@
1-
# Personal family tree data
2-
data/*.json
3-
data/*.tsv
4-
data/person/
5-
data/pruned/
6-
data/augment/
7-
data/photos/
8-
data/scrape/
9-
data/ai/
10-
11-
# Credentials (encrypted login info)
12-
data/credentials.json
13-
data/.credentials-key
1+
# All local data (family trees, SQLite db, credentials, photos, etc.)
2+
data/
143

154
# Browser automation (Chrome profile with auth cookies)
165
.browser/data/

0 commit comments

Comments
 (0)