From 88af565ca32d5cf3c07dd8b61dae51dbc4f42cc5 Mon Sep 17 00:00:00 2001 From: wintan1418 Date: Tue, 18 Aug 2026 20:13:30 +0100 Subject: [PATCH 1/2] Document that retry_on can't intercept process-death errors Active Job's retry_on and rescue_from only hook into exceptions raised while perform runs. ProcessPrunedError, ProcessExitError and ProcessMissingError are never raised inside the job: the process running it is already gone, and a different process records the error directly as a failed execution after the fact. People coming from other backends expect retry_on to cover this case and are surprised when it silently doesn't, so spell out why it can't work and point to the mechanisms that do: Mission Control and the fail_many_claimed subscription. Fixes #786 --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 54a09248..d9c8b484 100644 --- a/README.md +++ b/README.md @@ -669,6 +669,8 @@ end When a process dies without a clean shutdown (for example, `SIGKILL`ed by the OS or the container runtime because of memory limits), the jobs it was running can't be released back to their queues. Once another process notices the missing heartbeats and prunes the dead process's registration, its in-flight jobs are marked as failed with `SolidQueue::Processes::ProcessPrunedError`. Solid Queue deliberately doesn't retry these automatically: the job itself might be what's killing the process (for example, a job that exhausts the container's memory), and retrying it blindly would just kill the next worker too. +Note that Active Job's `retry_on` and `rescue_from` have no effect on these errors: they only intercept exceptions raised while your job's `perform` method runs, and `ProcessPrunedError` (as well as `ProcessExitError` and `ProcessMissingError`) is never raised inside the job. The process that was running the job is gone by then — the error is recorded directly as a failed execution by a *different* process, after the fact, so there's no job execution left for Active Job's retry machinery to hook into. To retry these jobs, act on the failed executions from the outside instead: manually, via [Mission Control — Jobs](https://github.com/rails/mission_control-jobs), or automatically, with a subscription like the one below. + If you know your jobs are idempotent and want to implement your own recovery policy, you can subscribe to the `fail_many_claimed.solid_queue` event, which includes the error and the affected job IDs in its payload: ```ruby From f5b5a9e3edd8c937bb7c434c22a28cca2f93d14c Mon Sep 17 00:00:00 2001 From: wintan1418 Date: Tue, 18 Aug 2026 20:31:40 +0100 Subject: [PATCH 2/2] Let bin/jobs check validate another env's config section without a database MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit bin/jobs check always read the config files through the current Rails env, so a CI run in the test env couldn't validate the production: section of config/recurring.yml — the one file whose env scoping is load-bearing. Worse, config_from's fallback made it silently validate nothing: with no test: key it took the whole file, treated production as a task with no schedule, dropped it, and reported the configuration valid. A new --env option threads the target env through to both config readers, so `bin/jobs check --env production` validates the section that will actually be deployed. Validation also crashed with a raw StatementInvalid backtrace when the solid_queue tables weren't reachable — instantiating RecurringTask needs its schema, and check already anticipates a missing database a few lines down, in the pool-size warning. Configuration now passes the scheduler raw [key, options] pairs instead of Active Record objects, which RecurringSchedule already knows how to wrap when the scheduler boots, so listing processes needs no database at all. Task-level validation still needs the schema; when it isn't available, check degrades to an explicit warning instead of failing, and still validates everything else. Fixes #780 --- lib/solid_queue/cli.rb | 6 +++++- lib/solid_queue/configuration.rb | 31 +++++++++++++++++++++++++------ test/unit/configuration_test.rb | 25 +++++++++++++++++++++++++ 3 files changed, 55 insertions(+), 7 deletions(-) diff --git a/lib/solid_queue/cli.rb b/lib/solid_queue/cli.rb index a52e9981..e7b2038c 100644 --- a/lib/solid_queue/cli.rb +++ b/lib/solid_queue/cli.rb @@ -35,7 +35,11 @@ def start SolidQueue::Supervisor.start(**options.symbolize_keys) end - desc :check, "Validates the Solid Queue configuration for the current Rails env without starting anything. Exits non-zero on errors." + desc :check, "Validates the Solid Queue configuration without starting anything. Exits non-zero on errors." + method_option :env, type: :string, + desc: "Environment section of the configuration files to validate (default: the current Rails env). " \ + "Lets a CI run in one env validate the section that will be deployed to another, e.g. --env production.", + banner: "ENVIRONMENT" def check configuration = SolidQueue::Configuration.new(**options.symbolize_keys) exit 1 unless configuration.check diff --git a/lib/solid_queue/configuration.rb b/lib/solid_queue/configuration.rb index b2153ede..9cefa812 100644 --- a/lib/solid_queue/configuration.rb +++ b/lib/solid_queue/configuration.rb @@ -100,6 +100,12 @@ def ensure_valid_recurring_tasks errors.add(:base, "Invalid recurring tasks:\n#{error_messages.join("\n")}") end + rescue ActiveRecord::ActiveRecordError + # Validating recurring tasks needs the solid_queue_recurring_tasks schema, and this + # environment has no usable database connection to load it from. Don't fail the whole + # check over that, but don't stay quiet about it either. + warnings.add(:base, "Warning: recurring tasks couldn't be validated because there's no usable " \ + "database connection. Run `bin/jobs check` with the Solid Queue database reachable to validate them.") end def warn_about_incorrectly_sized_database_pool @@ -156,6 +162,7 @@ def default_options { mode: ENV["SOLID_QUEUE_SUPERVISOR_MODE"] || :fork, standalone: true, + env: Rails.env, config_file: Rails.root.join(ENV["SOLID_QUEUE_CONFIG"] || DEFAULT_CONFIG_FILE_PATH), recurring_schedule_file: Rails.root.join(ENV["SOLID_QUEUE_RECURRING_SCHEDULE"] || DEFAULT_RECURRING_SCHEDULE_FILE_PATH), only_work: false, @@ -207,8 +214,8 @@ def dispatchers def schedulers return [] if skip_recurring_tasks? - if recurring_tasks.any? || dynamic_recurring_tasks_enabled? - [ Process.new(:scheduler, { recurring_tasks: recurring_tasks, **scheduler_options.with_defaults(SCHEDULER_DEFAULTS) }) ] + if recurring_task_definitions.any? || dynamic_recurring_tasks_enabled? + [ Process.new(:scheduler, { recurring_tasks: recurring_task_definitions, **scheduler_options.with_defaults(SCHEDULER_DEFAULTS) }) ] else [] end @@ -232,16 +239,24 @@ def dynamic_recurring_tasks_enabled? scheduler_options.fetch(:dynamic_tasks_enabled, SCHEDULER_DEFAULTS[:dynamic_tasks_enabled]) end + # Raw [ key, options ] pairs, with no database dependency: instantiating RecurringTask + # requires loading its schema, which isn't possible in an environment without a usable + # database connection. The scheduler wraps these in RecurringTask objects when it boots. + def recurring_task_definitions + @recurring_task_definitions ||= recurring_tasks_config.filter_map do |id, options| + [ id, options.merge(static: true) ] if options&.has_key?(:schedule) + end + end + def recurring_tasks - @recurring_tasks ||= recurring_tasks_config.map do |id, options| - RecurringTask.from_configuration(id, **options.merge(static: true)) if options&.has_key?(:schedule) - end.compact + @recurring_tasks ||= recurring_task_definitions.map { |definition| RecurringTask.wrap(definition) } end def processes_config @processes_config ||= config_from \ options.slice(:workers, :dispatchers, :scheduler).presence || options[:config_file], keys: [ :workers, :dispatchers, :scheduler ], + env: env, fallback: { workers: [ WORKER_DEFAULTS ], dispatchers: [ DISPATCHER_DEFAULTS ], @@ -251,10 +266,14 @@ def processes_config def recurring_tasks_config @recurring_tasks_config ||= begin - config_from options[:recurring_schedule_file] + config_from options[:recurring_schedule_file], env: env end end + def env + options[:env].presence || Rails.env + end + def config_from(file_or_hash, keys: [], fallback: {}, env: Rails.env) load_config_from(file_or_hash).then do |config| config = config[env.to_sym] ? config[env.to_sym] : config diff --git a/test/unit/configuration_test.rb b/test/unit/configuration_test.rb index 2a969a93..e32272de 100644 --- a/test/unit/configuration_test.rb +++ b/test/unit/configuration_test.rb @@ -283,6 +283,31 @@ class ConfigurationTest < ActiveSupport::TestCase assert configuration.valid? end + test "validate another environment's section of the config files with the env option" do + # Without env targeting, the production-only section is invisible from the test env + configuration = SolidQueue::Configuration.new(recurring_schedule_file: config_file_path(:recurring_with_production_only)) + assert_processes configuration, :scheduler, 0 + + configuration = SolidQueue::Configuration.new(recurring_schedule_file: config_file_path(:recurring_with_production_only), env: "production") + assert configuration.valid? + assert_processes configuration, :scheduler, 1 + + scheduler = configuration.configured_processes.detect { |process| process.kind == :scheduler }.instantiate + assert_has_recurring_task scheduler, key: "periodic_store_result", class_name: "StoreResultJob", schedule: "every second" + end + + test "check warns instead of erroring when recurring tasks can't be validated without a database" do + configuration = SolidQueue::Configuration.new(recurring_schedule_file: config_file_path(:recurring_with_invalid)) + SolidQueue::RecurringTask.stubs(:from_configuration) + .raises(ActiveRecord::StatementInvalid.new("Could not find table 'solid_queue_recurring_tasks'")) + + # The scheduler is still detected from the raw configuration, with no database involved + assert_processes configuration, :scheduler, 1 + + assert configuration.valid? + assert_match /recurring tasks couldn't be validated/, configuration.warnings.full_messages.join + end + test "reports an undersized database pool as a warning rather than an error" do configuration = SolidQueue::Configuration.new(workers: [ { queues: "background", threads: 50, polling_interval: 10 } ], skip_recurring: true)