Spectral Codex

This repository contains the working Astro project used to generate the Spectral Codex website.

Features

Content Management

• All content authored in MDX using the Content Layer API
• Quality scoring system (0-5 scale) drives site-wide content prioritization
• Comprehensive validation: frontmatter checks, cross-reference verification, geospatial boundary checking (Turf.js + FlatGeobuf), proximity-based duplicate detection (KDBush), image reference validation
• Automated excerpt generation for previews and listings
• Metadata index with automatic backlinks discovery from internal links
• Content linting and formatting via mdxlint with remark plugins
• Automatic redirect generation from formerIds frontmatter into Caddy config
• Media orphan detection for unreferenced images
• Link checker with SQLite persistence, per-domain rate limiting, auto-retry with staleness rechecking, digest-based change detection, and graceful shutdown handling

Image Handling

Content Layer Integration

• Experimental image loader treating individual images as first-class content with metadata extraction
• Automatic extraction of camera settings, GPS coordinates, and other EXIF data from images
• Automatic generation of data URI-encoded low-quality image placeholders (LQIPs)
• Custom remark plugin for advanced image layout (groups, carousels, aspect ratio handling)
• Hero image support with optional CSS-only image carousels

External Image Server

Astro's built-in image optimization works well for smaller sites, but this project has 8,000+ high-resolution source images. Processing them all during build leads to memory exhaustion and long build times. The solution: delegate image processing to an external service.

• Keep original image assets in the media folder specified in .env; high-quality JPG or lossless PNG format images at 2400+ pixels on the long edge are recommended, and the current standard is mostly based on 3,600 pixel JPGs saved at maximum quality in Lightroom
• imagor image server (MozJPEG build) handles on-demand resizing, format conversion, and quality adjustment
• Nginx reverse proxy with aggressive caching ensures images are only processed once
• URL-based transformations (e.g., /{signature}/fit-in/1200x800/filters:format(webp):quality(80)/path/to/image.jpg) allow flexible sizing without pre-generating variants
• Purpose-built typed imagor URL builder and HMAC-SHA256 signer (the unpic-imagor package) generates signed URLs at build time; signing plus rate limiting protects against cache-busting attacks
• Incremental cache warming with manifest tracking and remote SSH execution; only warms newly discovered URLs
• Docker Compose orchestration for easy deployment and updates

Interactive Maps

• React-based map component built with MapLibre, react-map-gl, and Protomaps
• Custom filter controls for adjusting what points are visible on the map
• Popups, clustering, filtering by objectives, and responsive design
• Administrative boundaries sourced from Overture Maps and converted to FlatGeobuf files for rendering on region maps
• Persistent storage of map data via IndexedDB
• Distance-based discovery via nearby locations, powered by kdbush spatial indexing for fast nearest-neighbor queries

Search & Discovery

• Client-side full-text search via Pagefind with a custom Astro integration (fork of shishkin/astro-pagefind), a modal interface via @pagefind/component-ui, keyboard shortcuts, and retina-ready thumbnails
• Related content recommendations via Transformers.js embeddings (MiniLM, MPNet, BGE-M3), USearch ANN indexing, and hybrid semantic + metadata ranking
• Hierarchical navigation through regions, themes, and series
• Client-side fuzzy 404 suggestions via fastest-levenshtein with substring-bonus scoring against a build-time content manifest; auto-redirects on near-exact matches

Timeline & Archives

• Chronological content browsing with yearly, monthly, and daily views
• Intelligent content deduplication across time periods based on created, updated, and visited dates
• Automatic highlight selection using quality scores
• Quality-based filtering with different thresholds for overview vs. detailed views

User Experience

• Native web components for interactive elements (dark mode toggle, reading progress, image carousels, pagination controls)
• Dark/light mode toggle with system preference detection and localStorage persistence
• Visual reading progress indicator for long-form content
• Loading progress bar during navigation
• Custom CJK character handling and language-specific styling
• Not fully internationalized; the goal of the project is to display multiple scripts on the same page without compromising aesthetics

SEO & Social

• Programmatic OG image generation via Satori and Sharp with multilingual font subsetting (Latin, CJK, Thai, Japanese), luminance-aware adaptive text color, and concurrent processing
• Hierarchical deterministic fallback system for entries without a featured image
• Digest-based caching; only regenerates when content or source image changes
• Comprehensive meta tags and structured data
• Custom sitemap integration with accurate per-URL lastmod dates derived from git commit history, so change dates survive content moves and rebuilds
• Full RSS feeds with server-side rendered MDX content via Astro's Container API
• Webmention support via webmention.io (optional, env-gated)

Analytics

• Self-hosted Umami site analytics monitoring web vitals performance metrics
• Custom event tracking for search queries, map filter changes, dark mode toggling, and image metadata interactions

Build & Deployment

Deployment is handled by custom TypeScript scripts. These are specific to this project's infrastructure but demonstrate some useful patterns. The full pipeline runs:

1. Content sync and validation
2. Redirect generation from former content IDs
3. Related content generation (semantic similarity)
4. OG image generation with Satori and Sharp
5. Astro production build
6. E2E smoke tests
7. Image cache warming manifest generation from built HTML
8. Media sync to remote storage
9. Static file transfer via rsync
10. OG image deployment
11. Caddy config and TLS cert sync with reload
12. CDN cache purge (Cloudflare)
13. New image cache warming

The image server is deployed separately and manually; it is only needed when image server code or Docker config changes.

License

This project is licensed under the MIT License. Feel free to use and adapt the code (but not the personal content specific to the project) for your own projects.