Skip to content

CHANGELOG.md

Latest commit

 

History

History
140 lines (108 loc) · 5.58 KB

CHANGELOG.md

File metadata and controls

140 lines (108 loc) · 5.58 KB

Changelog

All notable changes to this project will be documented in this file. The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

0.1.0 - 2025-11-09

Added

  • Production-Ready Release - First stable release of Singularity.Workflow
  • Database-Driven DAG Execution - PostgreSQL-based workflow orchestration
  • Workflow Lifecycle Management - Complete control over running workflows:
    • cancel_workflow_run/3 - Cancel running workflows with optional reason
    • list_workflow_runs/2 - Query workflows with filtering and pagination
    • retry_failed_workflow/3 - Retry failed workflows from point of failure
    • pause_workflow_run/2 - Pause workflow execution
    • resume_workflow_run/2 - Resume paused workflows
  • HTDAG Orchestration - Goal-driven workflow decomposition
  • Real-time Messaging - PostgreSQL NOTIFY for instant message delivery
  • Security Hardening - Safe string-to-atom conversion with validation
  • Code Quality - 0 Credo warnings, 100% formatted

Core Features

  • DAG Workflow Support with explicit dependencies
  • Parallel Execution for independent steps
  • Map Steps for bulk processing
  • Database-First Coordination via PostgreSQL + pgmq
  • Multi-Instance Scaling support
  • Comprehensive test coverage (678 tests)

Documentation

  • Complete API reference
  • HTDAG orchestration guide
  • Getting started guide
  • Architecture documentation
  • Phoenix integration examples

[1.0.1] - 2025-10-27

Added

  • Initial release of Singularity.Workflow - Elixir implementation of database-driven DAG execution
  • Complete feature parity including:
    • DAG Workflow Support: Define workflows with explicit dependencies between steps
    • Parallel Execution: Automatically execute independent steps in parallel
    • Map Steps: Execute the same step across multiple items (map/reduce pattern)
    • Dependency Merging: Steps can depend on multiple other steps
    • Database-First Coordination: PostgreSQL + pgmq for reliable task coordination
    • Multi-Instance Scaling: Multiple Singularity.Workflow instances can safely execute the same workflows
    • Visibility Timeout Pattern: Automatic retry if task executor crashes
    • Comprehensive Testing: 160+ tests covering all execution paths

Core Modules

  • Singularity.Workflow.Executor - Main entry point for workflow execution
  • Singularity.Workflow.FlowBuilder - Dynamic workflow construction API
  • Singularity.Workflow.DAG.WorkflowDefinition - DAG parsing and cycle detection
  • Singularity.Workflow.DAG.RunInitializer - Workflow initialization and state setup
  • Singularity.Workflow.DAG.TaskExecutor - Task execution and polling loop
  • Singularity.Workflow.StepState - Step state tracking (Ecto schema)
  • Singularity.Workflow.StepTask - Individual task tracking for map steps (Ecto schema)
  • Singularity.Workflow.WorkflowRun - Workflow execution tracking (Ecto schema)
  • Singularity.Workflow.StepDependency - DAG edge tracking (Ecto schema)

Database Features

  • PostgreSQL schema with efficient indices for query performance
  • pgmq extension integration for distributed task coordination
  • SQL functions for atomic operations:
    • complete_task() - Task completion with cascading to dependents
    • start_ready_steps() - Enqueue newly ready steps
    • start_tasks() - Start tasks with visibility timeout
    • fail_task() - Fail a task and cascade failure

Configuration

  • Configurable via environment variables:
    • DATABASE_URL - PostgreSQL connection
    • SINGULARITY_WORKFLOW_QUEUE_NAME - pgmq queue name (default: "singularity_workflow_queue")
    • SINGULARITY_WORKFLOW_VT - Visibility timeout in seconds (default: 300)
    • SINGULARITY_WORKFLOW_MAX_WORKERS - Max concurrent task executions (default: 10)

Documentation

  • GETTING_STARTED.md - Installation and first workflow tutorial
  • docs/ARCHITECTURE.md - Technical deep dive into internal design
  • CONTRIBUTING.md - Development guidelines and workflow
  • docs/API_REFERENCE.md - Complete API reference
  • docs/DYNAMIC_WORKFLOWS_GUIDE.md - Advanced workflow patterns
  • Security best practices documented in SECURITY.md

Development Tools

  • Code quality enforcement: mix quality runs all checks
    • mix format - Auto-formatting
    • mix credo --strict - Linting
    • mix dialyzer - Type checking
    • mix sobelow - Security analysis
    • mix test - Test suite with ExUnit
  • ExDoc integration for generated documentation

Known Limitations (v0.1.0)

  • Step definitions use Elixir module functions
    • Plan to support JSON workflow definitions in v0.2.0
  • Conditional step execution not yet supported
    • Plan for v0.2.0
  • No built-in loop/repeat step support
    • Use map steps as workaround
  • Limited to single-node deployments in this version
    • Multi-instance coordination planned for v0.2.0

Testing

  • 160+ comprehensive tests
  • Unit tests for individual components
  • Integration tests for complete workflows
  • Mock workflows for testing without side effects
  • Database transaction isolation for test safety

Performance

  • Task completion: O(1) via counter pattern (not row counting)
  • DAG cycle detection: O(V + E) depth-first search
  • Workflow startup latency: 50-500ms depending on database
  • Task execution throughput: 100-1000 tasks/second depending on complexity

[0.0.0] - Initial Development

Development version prior to public release.