Skip to content

Harden application resilience, job state, and data integrity #34

Description

@DenWin

Parent: #31. Coordinate with #18, #20, #21, and #7.

Current evidence

  • Database and filesystem changes span multiple steps without an explicit transaction/reconciliation model.
  • Conversion starts in the background after a redirect; failures log an error but may leave job state pending.
  • Promise.all/chunk failure semantics and partial output persistence are not specified.
  • Cleanup uses synchronous recursive deletion and a recurring timer in application bootstrap.
  • SQLite foreign-key enforcement and uniqueness constraints are not explicitly enabled/defined.
  • Several filesystem errors are logged and processing continues, creating possible DB/filesystem divergence.
  • Process cancellation, restart recovery, stuck-job detection, and graceful shutdown are not defined.

Required work

  • Define an explicit job state machine and allowed transitions.
  • Persist per-file and job failure reasons safely.
  • Define retry, idempotency, and duplicate-submission behavior.
  • Use transactions for coherent database state transitions.
  • Enable/verify relational constraints and migration invariants.
  • Define reconciliation for DB rows versus files after partial failure or restart.
  • Recover or mark interrupted/pending jobs on startup.
  • Add converter timeouts, cancellation, and graceful shutdown behavior.
  • Prevent unhandled promise rejections and perpetual pending states.
  • Make cleanup bounded, observable, retryable, and safe on unusual filesystems.
  • Define disk-full and database-busy/corruption behavior.
  • Add structured operational events and health/readiness distinctions.
  • Verify backup/restore consistency with active jobs.

Acceptance criteria

  • Job state transitions are specified and tested.
  • No failed conversion remains silently pending forever.
  • Restart and partial-failure recovery are deterministic.
  • Database/filesystem inconsistencies are detected and reconciled.
  • Constraints and migrations protect integrity.
  • Cleanup and shutdown cannot orphan uncontrolled subprocesses.
  • Fault-injection tests cover disk, DB, process, and restart failures.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions