Postmortem: Workshop Invitation Batch Failures (March–April 2026)

[mroderick]

Postmortem: Workshop Invitation Batch Failures (March–April 2026)

Date: 2026-04-10
Status: Resolved
Severity: High
Author: Morgan Roderick

---

Executive Summary

Between 2026-03-31 and 2026-04-09, workshop invitation batches failed repeatedly with the error "Member has already been taken." These failures prevented thousands of codebar members across 46 chapters from receiving workshop invitations.

The root cause traced back to a refactoring in May 2018 that accidentally removed deduplication logic from a database query. When the invitation logging system launched in April 2026, it surfaced a bug that had existed silently for seven years. Additional contributing issues were discovered during investigation, and all were addressed across five pull requests.

All three bugs have been resolved. The system now reliably sends invitations across all chapters.

---

Why This Went Unreported for Seven Years

This section explains why a bug introduced in May 2018 remained hidden until April 2026.

The Bug Chain

The "Member has already been taken" error requires two separate bugs to interact:

1. Bug 1 (May 2018): Missing .distinct in queries produces duplicate member rows
2. Bug 3 (April 2026): InvitationLogEntry uniqueness constraint rejects duplicates

Either bug alone does not produce the error:

Scenario	Bug 1 (duplicate rows)	Bug 3 (uniqueness constraint)	Both together
Single audience ("students" only)	Member appears twice in iteration	Each member gets one entry	No error — find_or_create_by handles duplicates
"Everyone" audience	Member in both student and coach lists	Different roles	Error manifests

Why Single-Audience Sends Didn't Trigger the Bug

When sending invitations to students only:

• A member subscribed to two student groups appears twice in the iteration
• First invitation succeeds (creates WorkshopInvitation)
• Second invitation uses find_or_create_by → finds existing invitation, returns it
• No error occurs — silently succeeds

When sending to everyone (both audiences):

• A member subscribed to both a student group and a coach group appears in both lists
• First role processes fine
• Second role triggers the uniqueness validation on InvitationLogEntry
• Error manifests

Additional Factor: UI Workflow

The admin interface at app/views/admin/workshops/send_invites.html.haml explicitly encourages sending to one audience at a time:

%p
  If you click "Students" now, you can come back and send them to the coaches at a separate time, and vice versa.

The "Everybody" button exists but sees less use than the individual audience buttons. This reduced the frequency of the trigger condition.

---

Timeline

Date	Event
2017-01	Original code includes .uniq for deduplication (commit 448acc7c)
2018-05	Refactoring removes .uniq, introducing the dormant bug (commit d7a463da)
2026-03-31	First failed batch observed in production (Berlin chapter)
2026-04-07	Issue #2542 reported by chapter organiser
2026-04-07	PR #2546 ships invitation logging system
2026-04-08	PR #2551 adds .distinct to fix duplicate member processing
2026-04-08	PR #2553 adds unique index to subscriptions table
2026-04-09	PR #2556 fixes retry logic in InvitationLogger
2026-04-09	PR #2558 refactors to idiomatic Rails pattern

---

Root Cause Analysis

Three distinct bugs produced the same error message. This section explains each.

Bug 1: Missing Query Deduplication

Affected file: app/models/invitation_manager.rb

The chapter_students and chapter_coaches methods query the database using a JOIN on the subscriptions table:

def chapter_students(chapter)
  Member.in_group(chapter.groups.students)
end

When a member subscribes to multiple groups within a chapter, the JOIN produces duplicate rows. In the London chapter, 336 members had this issue.

The original code in January 2017 used .uniq to deduplicate:

workshop.chapter.groups.students.map(&:members).flatten.uniq

The May 2018 refactoring extracted this to a scope but omitted the deduplication:

scope :in_group, ->(group) { not_banned.joins(:groups).merge(group) }

This bug lay dormant for seven years because invitation batches typically targeted single audiences ("students" OR "coaches"). The bug only surfaced when sending to "everyone" (both audiences), which processes more members and triggers the duplicate rows.

Fix applied: PR #2551 adds .distinct to the query methods.

---

Bug 2: Subscription Race Conditions

Affected table: subscriptions

The model validation validates :group, uniqueness: { scope: :member_id } prevents duplicate subscriptions at the application level. However, it cannot prevent race conditions where two concurrent requests both pass validation before either saves.

Analysis of production data revealed 25 unique member/group combinations with duplicate subscriptions created within 3 milliseconds to 2.5 seconds of each other. These spanned 2015 to 2023.

Fix applied: PR #2553 adds a database-level unique index on (member_id, group_id) and cleans up 31 existing duplicate records.

---

Bug 3: Cross-Status Retry in InvitationLogger

Affected file: app/services/invitation_logger.rb

The InvitationLogEntry model validates uniqueness on (member_id, invitation_type, invitation_id). Status is NOT part of the constraint, meaning a member can have both a "success" entry AND a "failure" entry for the same invitation.

The original retry logic used:

@log.entries.find_or_initialize_by(member:, invitation:, status:)

When an entry existed with status failed but the code requested status success, the query returned the existing (wrong-status) entry. The subsequent save attempt triggered the uniqueness validation error.

Fix applied: PR #2556 and #2558 refactor to check for existing entries without including status in the query:

def find_or_build_entry(member, invitation, status)
  existing_entry = @log.entries.find_by(member: member, invitation: invitation)
  return existing_entry if existing_entry

  @log.entries.new(member: member, invitation: invitation, status: status)
end

Community review (olleolleolle) suggested using find_or_create_by with a block, which was implemented in the final version for cleaner, more idiomatic Rails code.

---

Impact

Chapters Affected

46 of 54 chapters (85%) had members with duplicate subscriptions.

Chapter	Members Affected
London	336
West London	79
Brighton	57
South London	57
Barcelona	55
Edinburgh	37
Berlin	35
(39 more chapters)	~600 total

Failed Batches Observed

• Berlin: 3 failed attempts (99, 1, 1 members processed)
• Brighton: 3 failed attempts (493 members before failure)
• London: 3 failed attempts (517 members before failure)

In each failed batch, the remaining members never received invitations.

---

What Went Well

1. Systematic root cause isolation — The team methodically separated the three bugs despite identical error messages.

2. Community contributions — Review feedback improved the final fix, demonstrating the value of open-source collaboration.

3. Production data analysis — Queries against production identified the full scope: 1,200 affected members across 46 chapters.

4. Complementary fixes — Each PR addressed a distinct aspect, ensuring the solution is robust across different scenarios.

---

Prevention

Query Deduplication Tests

Add test coverage for methods that return members from JOIN queries. Ensure tests cover the case where a member appears in multiple groups.

Action: Review existing specs for chapter_students and chapter_coaches, add edge case coverage.

Database-Level Uniqueness

All polymorphic associations with user-facing uniqueness constraints should have database-level unique indexes.

Action: Audit models for validates :uniqueness without corresponding database indexes.

Post-Deploy Verification

Include verification queries in deployment runbooks to confirm expected state after schema or model changes.

Example query:

-- Verify no duplicate subscriptions
SELECT member_id, group_id, COUNT(*) as cnt
FROM subscriptions
GROUP BY member_id, group_id
HAVING COUNT(*) > 1;

---

Acknowledgments

Thank you to all contributors who helped diagnose and fix these issues:

• Olle Jonsson (olleolleolle) — Code review that improved the final solution
• Till Till Klampaeckel (till) - Pairing and analysis
• The codebar chapter organisers who reported the problem and provided data
• All volunteers who maintain this open-source project

---

Related Pull Requests

• PR #2546: Add database-backed invitation logging
• PR #2551: Fix deduplicate members in chapter_students and chapter_coaches
• PR #2553: Add unique index on subscriptions
• PR #2556: Handle duplicate InvitationLogEntry on retry
• PR #2558: Handle duplicate InvitationLogEntry on retry (v2)

[mroderick]

I suspect that there might still be a gremlin in the system. For the upcoming workshop in Berlin, I want to double check the invitations in the database against the chapter members, to double check that even with the invitation log bugs we experienced, that it still can and has sent out invitations to everyone eligible.

Further evidence: https://app.rollbar.com/a/codebar-production/fix/item/codebar-production/674#detail

I'll take a closer look over the weekend

[mroderick]

That issue has also been fixed. All is well with invitations now