Skip to content

Conversation

@dimitri-yatsenko
Copy link
Member

Summary

Stop tracking llms-full.txt in git and auto-generate it during deployment with version metadata.

Changes

Generator Script (scripts/gen_llms_full.py)

  • Added get_git_info() function to extract commit hash and branch name
  • Updated header template to include timestamp, commit, and branch
  • Modified generate_llms_full() to populate header with current build info

Build Process (docker-compose.yaml)

  • Added python scripts/gen_llms_full.py to BUILD mode before mkdocs build
  • Ensures file is generated fresh on every deployment with current git metadata

Version Control

  • Added src/llms-full.txt and site/llms-full.txt to .gitignore
  • Removed src/llms-full.txt from git tracking (30,871 lines deleted)

Example Output

# DataJoint Documentation (Full)

Generated: 2026-01-16 20:39:01 UTC
Commit: dc021ef
Branch: main

> DataJoint is a Python framework for building...

Benefits

  1. Cleaner git history - No more 30K line diffs on doc changes
  2. No merge conflicts - Generated files won't conflict across branches
  3. Always fresh - File regenerates with every build
  4. Traceable - Version metadata shows when/where it was generated
  5. Standard practice - Follows documentation best practices

Testing

  • Generator script runs successfully
  • Metadata includes timestamp, commit hash, and branch
  • File is properly excluded from git tracking
  • Local generation produces correct output (31,555 lines)
  • CI/CD build succeeds (will verify after merge)
  • Deployed file accessible with metadata

Notes

  • src/llms.txt (85-line index) remains committed ✓
  • No changes to GitHub Actions workflow needed
  • Backwards compatible - same URL, same content structure
  • File will be auto-generated during next deployment

- Add version metadata (timestamp, commit, branch) to generated file
- Generate llms-full.txt during docker-compose BUILD mode
- Exclude llms-full.txt from git tracking (30,869+ lines removed)
- Add to .gitignore to prevent future commits

Benefits:
- Cleaner git history without 30K line diffs on doc changes
- No merge conflicts from generated files
- Always fresh with current build metadata
- Follows documentation best practices
@MilagrosMarin MilagrosMarin merged commit 0589aec into main Jan 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants