toast: pluggable / multiple TOAST support#39
Draft
gburd wants to merge 11 commits into
Draft
Conversation
- Hourly upstream sync from postgres/postgres (24x daily) - AI-powered PR reviews using AWS Bedrock Claude Sonnet 4.5 - Multi-platform CI via existing Cirrus CI configuration - Cost tracking and comprehensive documentation Features: - Automatic issue creation on sync conflicts - PostgreSQL-specific code review prompts (C, SQL, docs, build) - Cost limits: $15/PR, $200/month - Inline PR comments with security/performance labels - Skip draft PRs to save costs Documentation: - .github/SETUP_SUMMARY.md - Quick setup overview - .github/QUICKSTART.md - 15-minute setup guide - .github/PRE_COMMIT_CHECKLIST.md - Verification checklist - .github/docs/ - Detailed guides for sync, AI review, Bedrock See .github/README.md for complete overview Complete Phase 3: Windows builds + fix sync for CI/CD commits Phase 3: Windows Dependency Build System - Implement full build workflow (OpenSSL, zlib, libxml2) - Smart caching by version hash (80% cost reduction) - Dependency bundling with manifest generation - Weekly auto-refresh + manual triggers - PowerShell download helper script - Comprehensive usage documentation Sync Workflow Fix: - Allow .github/ commits (CI/CD config) on master - Detect and reject code commits outside .github/ - Merge upstream while preserving .github/ changes - Create issues only for actual pristine violations Documentation: - Complete Windows build usage guide - Update all status docs to 100% complete - Phase 3 completion summary All three CI/CD phases complete (100%): ✅ Hourly upstream sync with .github/ preservation ✅ AI-powered PR reviews via Bedrock Claude 4.5 ✅ Windows dependency builds with smart caching Cost: $40-60/month total See .github/PHASE3_COMPLETE.md for details Fix sync to allow 'dev setup' commits on master The sync workflow was failing because the 'dev setup v19' commit modifies files outside .github/. Updated workflows to recognize commits with messages starting with 'dev setup' as allowed on master. Changes: - Detect 'dev setup' commits by message pattern (case-insensitive) - Allow merge if commits are .github/ OR dev setup OR both - Update merge messages to reflect preserved changes - Document pristine master policy with examples This allows personal development environment commits (IDE configs, debugging tools, shell aliases, Nix configs, etc.) on master without violating the pristine mirror policy. Future dev environment updates should start with 'dev setup' in the commit message to be automatically recognized and preserved. See .github/docs/pristine-master-policy.md for complete policy See .github/DEV_SETUP_FIX.md for fix summary Optimize CI/CD costs by skipping builds for pristine commits Add cost optimization to Windows dependency builds to avoid expensive builds when only pristine commits are pushed (dev setup commits or .github/ configuration changes). Changes: - Add check-changes job to detect pristine-only pushes - Skip Windows builds when all commits are dev setup or .github/ only - Add comprehensive cost optimization documentation - Update README with cost savings (~40% reduction) Expected savings: ~$3-5/month on Windows builds, ~$40-47/month total through combined optimizations. Manual dispatch and scheduled builds always run regardless.
Introduces a pluggable TOAST (The Oversized-Attribute Storage Technique) API that allows custom TOAST implementations per column/type. Adds new TsrRoutine structure with function pointers for toast operations, enabling extensible TOAST mechanics beyond the traditional heap-based implementation. Co-authored-by: Greg Burd <greg@burd.me>
Refactors the existing heap TOAST implementation to use the new pluggable TOAST API, demonstrating how the default toaster integrates with the new framework. Co-authored-by: Greg Burd <greg@burd.me>
Implements iterator-based detoasting for streaming large TOAST values without loading entire objects into memory at once. Adds DetoastIterator and FetchDatumIterator structures for incremental decompression. Co-authored-by: Greg Burd <greg@burd.me>
Provides a minimal example TOAST implementation for testing and demonstration purposes. Shows how to implement custom toasters using the pluggable API. Co-authored-by: Greg Burd <greg@burd.me>
Adds reconstruct_toast_function callback to enable logical replication to properly handle custom TOAST formats. Necessary for decoding and transmitting toasted data in logical replication streams. Co-authored-by: Greg Burd <greg@burd.me>
Implements an appendable TOAST storage mechanism optimized for append-only bytea columns, demonstrating advanced custom toaster capabilities. Co-authored-by: Greg Burd <greg@burd.me>
Major architectural change allowing each relation to have multiple TOAST tables, one per toaster. Replaces single reltoastrelid with arrays reltoasterids and reltoastrelids in pg_class. Updates catalog, relcache, vacuum, and autovacuum to handle multiple TOAST relations. Co-authored-by: Greg Burd <greg@burd.me>
Updates del_toast_function signature to accept Relation parameter, enabling toasters to access relation metadata during deletion operations. Co-authored-by: Greg Burd <greg@burd.me>
Introduces VARATT_CUSTOM structure for custom TOAST pointer formats, allowing toasters to define their own external storage metadata beyond the standard varatt_external format. Co-authored-by: Greg Burd <greg@burd.me>
github-actions
Bot
force-pushed
the
master
branch
3 times, most recently
from
15913cb to
06a2a69
Compare
gburd
force-pushed
the
master
branch
2 times, most recently
from
3b49b2a to
0b3e646
Compare
github-actions
Bot
force-pushed
the
master
branch
3 times, most recently
from
e495986 to
d7cd5d6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft PR opened to enable OCR AI review on the
multi-toastbranch. Title/description are provisional — rename before publishing to pgsql-hackers.