WIP N+1 detector gem Prosopite (replacing gem Bullet)

Gemfile

@@ -63,7 +63,7 @@ gem "wicked" # Multi-step form wizard for Rails
 
 group :development, :test do
   gem "brakeman" # Security inspection
-  gem "bullet" # Detect and fix N+1 queries
+  gem "prosopite" # N+1 query detection via SQL pattern analysis
   gem "byebug", platforms: %i[mri mingw x64_mingw] # Debugger console
   gem "dotenv-rails" # Environment variable management
   gem "erb_lint", require: false # ERB linter

Gemfile.lock

@@ -115,9 +115,6 @@ GEM
     bugsnag (6.28.0)
       concurrent-ruby (~> 1.0)
     builder (3.3.0)
-    bullet (8.1.0)
-      activesupport (>= 3.0.0)
-      uniform_notifier (~> 1.11)
     bundler-audit (0.9.3)
       bundler (>= 1.2.0)
       thor (~> 1.0)
@@ -423,6 +420,7 @@ GEM
       actionpack (>= 7.1)
     prettyprint (0.2.0)
     prism (1.9.0)
+    prosopite (2.1.2)
     pry (0.16.0)
       coderay (~> 1.1)
       method_source (~> 1.0)
@@ -665,7 +663,6 @@ GEM
     unicode-display_width (3.2.0)
       unicode-emoji (~> 4.1)
     unicode-emoji (4.2.0)
-    uniform_notifier (1.18.0)
     useragent (0.16.11)
     view_component (4.2.0)
       actionview (>= 7.1.0)
@@ -710,7 +707,6 @@ DEPENDENCIES
   blueprinter
   brakeman
   bugsnag
-  bullet
   bundler-audit
   byebug
   capybara
@@ -754,6 +750,7 @@ DEPENDENCIES
   pg_query
   pghero
   pretender
+  prosopite
   pry
   pry-byebug
   puma (~> 7.0)

PROSOPITE_TODO.md

@@ -0,0 +1,145 @@
+# Prosopite N+1 Query Issues
+
+Issues detected by Prosopite during test suite run. Fix by adding eager loading (`includes`, `preload`) or restructuring queries.
+
+## High Priority (20+ occurrences)
+
+### app/decorators/casa_case_decorator.rb:34
+- **Method:** `map` in decorator
+- **Likely fix:** Add `includes` for associated records being accessed in iteration
+
+### app/models/user.rb:84
+- **Method:** `create_preference_set`
+- **Query:** `SELECT "preference_sets".* FROM "preference_sets" WHERE "preference_sets"."user_id" = $1`
+- **Likely fix:** Check if preference_set already loaded before querying
+
+### app/models/concerns/api.rb:11
+- **Method:** `initialize_api_credentials`
+- **Query:** `SELECT "api_credentials".* FROM "api_credentials" WHERE "api_credentials"."user_id" = $1`
+- **Likely fix:** Check if api_credentials already loaded before querying
+
+### app/lib/importers/file_importer.rb:50
+- **Method:** `create_user_record`
+- **Queries:** Multiple user lookups during import
+- **Likely fix:** Batch lookups or use `Prosopite.pause` for intentional import operations
+
+## Medium Priority (10-19 occurrences)
+
+### app/validators/user_validator.rb:56
+- **Method:** `validate_uniqueness`
+- **Query:** `SELECT "users".* FROM "users" WHERE "users"."type" = $1 AND "users"."email" = $2`
+- **Likely fix:** Consider if validation is necessary during bulk operations
+
+### app/lib/importers/supervisor_importer.rb:47
+- **Method:** `block in assign_volunteers`
+- **Query:** `SELECT "users".* FROM "users" INNER JOIN "supervisor_volunteers"...`
+- **Likely fix:** Preload volunteers before assignment loop
+
+### app/lib/importers/supervisor_importer.rb:51
+- **Method:** `block in assign_volunteers`
+- **Query:** `SELECT "users".* FROM "users" WHERE "users"."id" = $1`
+- **Likely fix:** Batch load users by ID before iteration
+
+### app/controllers/case_contacts/form_controller.rb:156
+- **Method:** `block in create_additional_case_contacts`
+- **Likely fix:** Eager load case contact associations
+
+## Lower Priority (5-9 occurrences)
+
+### app/models/court_date.rb:32
+- **Method:** `associated_reports`
+- **Likely fix:** Add `includes` for court report associations
+
+### app/lib/importers/supervisor_importer.rb:23
+- **Method:** `block in import_supervisors`
+- **Query:** `SELECT "users".* FROM "users" WHERE "users"."type" = $1 AND "users"."email" = $2`
+- **Likely fix:** Batch check existing supervisors before import loop
+
+## Lower Priority (2-4 occurrences)
+
+### app/lib/importers/file_importer.rb:57
+- **Method:** `email_addresses_to_users`
+- **Likely fix:** Batch load users by email
+
+### app/lib/importers/case_importer.rb:41
+- **Method:** `create_casa_case`
+- **Likely fix:** Preload or batch casa_case lookups
+
+### app/decorators/contact_type_decorator.rb:14
+- **Method:** `last_time_used_with_cases`
+- **Likely fix:** Eager load case associations
+
+### app/datatables/reimbursement_datatable.rb:25
+- **Method:** `block in data`
+- **Query:** `SELECT "addresses".* FROM "addresses" WHERE "addresses"."user_id" = $1`
+- **Likely fix:** Add `includes(:address)` to reimbursement query
+
+### app/services/volunteer_birthday_reminder_service.rb:7
+- **Method:** `block in send_reminders`
+- **Likely fix:** Eager load volunteer associations
+
+### app/models/contact_topic.rb:27
+- **Method:** `block in generate_for_org!`
+- **Likely fix:** Batch operations or use `Prosopite.pause` for setup
+
+### app/models/casa_org.rb:82
+- **Method:** `user_count`
+- **Likely fix:** Use counter cache or single count query
+
+### app/models/casa_org.rb:62
+- **Method:** `case_contacts_count`
+- **Likely fix:** Use counter cache or single count query
+
+### app/lib/importers/volunteer_importer.rb:23
+- **Method:** `block in import_volunteers`
+- **Likely fix:** Batch check existing volunteers before import loop
+
+### app/lib/importers/case_importer.rb:20
+- **Method:** `block in import_cases`
+- **Likely fix:** Batch check existing cases before import loop
+
+### app/controllers/case_contacts/form_controller.rb:26
+- **Method:** `block (2 levels) in update`
+- **Likely fix:** Eager load contact associations
+
+## Single Occurrence
+
+### app/services/missing_data_export_csv_service.rb:40
+- **Method:** `full_data`
+
+### app/policies/contact_topic_answer_policy.rb:18
+- **Method:** `create?`
+
+### app/models/casa_case.rb:152
+- **Method:** `next_court_date`
+
+### app/models/all_casa_admins/casa_org_metrics.rb:16
+- **Method:** `map`
+
+### config/initializers/sent_email_event.rb:7
+- **Method:** `block in <top (required)>`
+- **Query:** `SELECT "casa_orgs".* FROM "casa_orgs" WHERE "casa_orgs"."id" = $1`
+- **Note:** Initializer callback - consider caching org lookup
+
+## Notes
+
+- **Importers:** Many N+1s occur in import code. Consider wrapping entire import operations in `Prosopite.pause { }` if the N+1 pattern is intentional for per-record processing, or batch-load records before iteration.
+
+- **Decorators:** Add `includes` at the controller/query level before passing to decorators.
+
+- **Callbacks:** User model callbacks (`create_preference_set`, `initialize_api_credentials`) fire on each create. Consider if these can be optimized or if the N+1 is acceptable for single-record operations.
+
+## How to Fix
+
+```ruby
+# Before (N+1)
+users.each { |u| u.orders.count }
+
+# After (eager loading)
+users.includes(:orders).each { |u| u.orders.count }
+
+# For intentional batch operations
+Prosopite.pause do
+  records.each { |r| process_individually(r) }
+end
+```

config/environments/development.rb

@@ -68,12 +68,9 @@
   # Raises error for missing translations.
   # config.i18n.raise_on_missing_translations = true
 
-  config.after_initialize do
-    Bullet.enable = true
-    Bullet.console = true
-    Bullet.rails_logger = true
-    Bullet.bullet_logger = true
-  end
+  # Prosopite N+1 query detection
+  config.prosopite_enabled = true
+  config.prosopite_min_n_queries = 5  # More lenient for development
 
   # Annotate rendered view with file names.
   config.action_view.annotate_rendered_view_with_filenames = true

config/environments/production.rb

@@ -108,4 +108,8 @@
   # ]
   # Skip DNS rebinding protection for the default health check endpoint.
   # config.host_authorization = { exclude: ->(request) { request.path == "/up" } }
+
+  # Prosopite N+1 query detection (disabled by default in production)
+  config.prosopite_enabled = ENV.fetch("PROSOPITE_ENABLED", "false") == "true"
+  config.prosopite_min_n_queries = ENV.fetch("PROSOPITE_MIN_N_QUERIES", "10").to_i
 end

config/environments/test.rb

@@ -63,13 +63,9 @@
   # Raises error for missing translations.
   config.i18n.raise_on_missing_translations = true
 
-  config.after_initialize do
-    Bullet.enable = true
-    Bullet.console = true
-    Bullet.bullet_logger = true
-    Bullet.rails_logger = true
-    # Bullet.raise = true # TODO https://github.com/rubyforgood/casa/issues/2441
-  end
+  # Prosopite N+1 query detection
+  config.prosopite_enabled = true
+  config.prosopite_min_n_queries = 2  # Stricter for tests
 
   # Annotate rendered view with file names.
   # config.action_view.annotate_rendered_view_with_filenames = true

config/initializers/prosopite.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+# Only enable Rack middleware if Prosopite is configured on
+if Rails.configuration.respond_to?(:prosopite_enabled) && Rails.configuration.prosopite_enabled
+  require "prosopite/middleware/rack"
+  Rails.configuration.middleware.use(Prosopite::Middleware::Rack)
+end
+
+Rails.application.config.after_initialize do
+  # Core settings
+  Prosopite.enabled = Rails.configuration.respond_to?(:prosopite_enabled) &&
+                      Rails.configuration.prosopite_enabled
+
+  # Minimum repeated queries to trigger detection (default: 2)
+  Prosopite.min_n_queries = Rails.configuration.respond_to?(:prosopite_min_n_queries) ?
+                            Rails.configuration.prosopite_min_n_queries : 2
+
+  # Logging options
+  Prosopite.rails_logger = true                    # Log to Rails.logger
+  Prosopite.prosopite_logger = Rails.env.development?  # Log to log/prosopite.log
+end

spec/.prosopite_ignore

@@ -0,0 +1,7 @@
+# Directories to exclude from Prosopite N+1 scanning
+# Remove paths as you fix N+1s in each area
+#
+# Example entries:
+# spec/features
+# spec/system
+# spec/requests/legacy

spec/support/prosopite.rb

@@ -0,0 +1,69 @@
+# frozen_string_literal: true
+
+return unless defined?(Prosopite)
+
+# Test configuration
+Prosopite.enabled = true
+Prosopite.raise = false  # Log only, don't fail specs
+Prosopite.rails_logger = true
+Prosopite.prosopite_logger = true
+
+# Allowlist for known acceptable N+1 patterns (e.g., test matchers)
+Prosopite.allow_stack_paths = [
+  "shoulda/matchers/active_record/validate_uniqueness_of_matcher.rb",
+  "shoulda/matchers/active_model/validate_presence_of_matcher.rb",
+  "shoulda/matchers/active_model/validate_inclusion_of_matcher.rb",
+  "shoulda/matchers/active_model/allow_value_matcher.rb"
+]
+
+# Optional: Load ignore list from file for gradual rollout
+PROSOPITE_IGNORE = if File.exist?("spec/.prosopite_ignore")
+  File.read("spec/.prosopite_ignore").lines.map(&:chomp).reject(&:empty?)
+else
+  []
+end
+
+# Monkey-patch FactoryBot to pause during factory creation
+# Prevents false positives from factory callbacks
+if defined?(FactoryBot)
+  module FactoryBot
+    module Strategy
+      class Create
+        alias_method :original_result, :result
+
+        def result(evaluation)
+          if defined?(Prosopite) && Prosopite.enabled?
+            Prosopite.pause { original_result(evaluation) }
+          else
+            original_result(evaluation)
+          end
+        end
+      end
+    end
+  end
+end
+
+# RSpec integration
+RSpec.configure do |config|
+  config.around do |example|
+    if use_prosopite?(example)
+      Prosopite.scan { example.run }
+    else
+      original_enabled = Prosopite.enabled?
+      Prosopite.enabled = false
+      example.run
+      Prosopite.enabled = original_enabled
+    end
+  end
+end
+
+def use_prosopite?(example)
+  # Explicit metadata takes precedence
+  return false if example.metadata[:disable_prosopite]
+  return true if example.metadata[:enable_prosopite]
+
+  # Check against ignore list
+  PROSOPITE_IGNORE.none? do |path|
+    File.fnmatch?("./#{path}/*", example.metadata[:rerun_file_path].to_s)
+  end
+end