Bug: Pop recommender undercounts item popularity due to duplicate items in batches

[rkawajiri]

Bug: Pop recommender undercounts item popularity due to duplicate items in batches

Description

The Pop (Popularity-based) recommender has a critical bug in the calculate_loss method that causes it to undercount item popularity when the same item appears multiple times within a single batch. This leads to significantly inaccurate popularity scores and affects its effectiveness as a baseline model.

Bug Location

File: recbole/model/general_recommender/pop.py
Lines: 45-47

Root Cause

The current implementation uses PyTorch tensor indexing to update item counts:

def calculate_loss(self, interaction):
    item = interaction[self.ITEM_ID]
    self.item_cnt[item, :] = self.item_cnt[item, :] + 1  # BUG HERE

Problem: When using assignment with duplicate indices (tensor[indices] = values), PyTorch does NOT accumulate values for duplicate indices. Instead, each assignment for the same index overwrites the previous one, so only the LAST occurrence takes effect.

Example Demonstrating the Bug

import torch

n_items = 10
item_cnt = torch.zeros(n_items, 1, dtype=torch.long)

# Batch where item 3 appears 3 times
items = torch.tensor([3, 5, 3, 7, 3, 5])

# Buggy approach
item_cnt[items, :] = item_cnt[items, :] + 1

# Result:
# item_cnt[3] = 1  ❌ (should be 3!)
# item_cnt[5] = 1  ❌ (should be 2!)
# item_cnt[7] = 1  ✓ (correct)

What happens internally:

1. item_cnt[3] = 0 + 1 = 1
2. item_cnt[5] = 0 + 1 = 1
3. item_cnt[3] = 1 + 1 = 2 (overwrites step 1)
4. item_cnt[7] = 0 + 1 = 1
5. item_cnt[3] = 1 + 1 = 2 (overwrites step 3)
6. item_cnt[5] = 1 + 1 = 2 (overwrites step 2)

Final result: All items have count = 1, regardless of how many times they appeared!

Impact

The bug's severity increases with:

• Larger batch sizes (more likely to have duplicates)
• Lower sparsity datasets (items appear more frequently)
• Popular items (appear in more batches)

Quantitative Impact Analysis

Results from ml-100k dataset with batch_size=2048:

• Total items: 1,683
• Items undercounted: 1,663 out of 1,682 items with interactions (98.9%)
• Total undercount: 46.75% of true item counts (86,061 buggy vs 161,616 correct)
• Average undercount per item: 44.92 occurrences
• Maximum undercount for single item: 421 occurrences
• Average percent undercount: 38.33%
• Maximum percent undercount: 84.20%

Key Finding: The most popular item was counted as 79 times (buggy) when it actually appeared 500 times (84% undercount!)

Impact on Top-K Rankings

The bug fundamentally distorts the popularity ranking:

Most Popular Item Example:

• Buggy ranking: May not be Config module #1 (counted only 79 times)
• Correct ranking: Should be Config module #1 (appears 500 times)
• This item's true popularity is 6.3x higher than the buggy count suggests

Expected Impact on Top-20:
With 98.9% of items affected and an average 38% undercount, the Top-20 ranking is likely to have:

• Significant position changes: Items with higher duplicate rates in batches will move up in ranking
• New items entering Top-20: Items that were undercounted more severely may now appear in Top-20
• Items dropping out: Items that were overcounted relatively (appeared mostly once per batch) will drop

Research Implications:

• Comparison studies using Pop as baseline are comparing against incorrect popularity rankings
• The bug makes Pop appear less effective than it actually is
• Any paper reporting "X% improvement over Pop" may be reporting inflated gains

Steps to Reproduce

1. Create a simple test script:

import torch
from recbole.model.general_recommender import Pop
from recbole.config import Config

# Setup
config = Config(model='Pop', dataset='ml-100k')
model = Pop(config, None)
model.n_items = 10
model.item_cnt = torch.zeros(10, 1, dtype=torch.long, device='cpu', requires_grad=False)

# Create batch with duplicate item 3
interaction = {config['ITEM_ID_FIELD']: torch.tensor([3, 3, 3, 5, 5])}
model.calculate_loss(interaction)

# Check counts
print(f"Item 3 count: {model.item_cnt[3].item()}")  # Prints 1, should be 3
print(f"Item 5 count: {model.item_cnt[5].item()}")  # Prints 1, should be 2

2. Observe that duplicate items are only counted once

Proposed Fix

Use scatter_add_() for proper atomic accumulation:

def calculate_loss(self, interaction):
    item = interaction[self.ITEM_ID]
    # Use scatter_add_ for proper accumulation of duplicate items
    ones = torch.ones_like(item).unsqueeze(1).to(dtype=self.item_cnt.dtype)
    self.item_cnt.scatter_add_(0, item.unsqueeze(1), ones)

    self.max_cnt = torch.max(self.item_cnt, dim=0)[0]

    return torch.nn.Parameter(torch.zeros(1)).to(self.device)

Why this works:

• scatter_add_() is an atomic accumulation operation designed for this use case
• It properly handles duplicate indices by accumulating all values
• This pattern is already used in other RecBole models (RepeatNet:256, GRU4RecCPR:381-385, SASRecCPR:389-395)

Verification

The fix has been verified with unit tests covering:

• Duplicate items in single batch
• Accumulation across multiple batches
• Large batches with many duplicates
• CPU and CUDA compatibility

All tests pass ✓

System Information

• RecBole Version: 1.2.1
• PyTorch Version: 2.7.1+cpu
• Python Version: 3.12.3
• OS: Linux
• CUDA Version: N/A (CPU only)

Additional Context

This bug has likely gone unnoticed because:

1. Recommendation data is typically sparse (duplicates are less common)
2. Pop is primarily used as a baseline (not scrutinized heavily)
3. The model still produces rankings (just incorrect ones)
4. Tests only verify the model runs, not accuracy of counts

However, for research comparing against Pop as a baseline, this bug significantly affects the validity of comparisons.

Related Code

Similar correct implementations using scatter_add_ in RecBole:

• recbole/model/sequential_recommender/repeatnet.py:256
• recbole/model/sequential_recommender/gru4reccpr.py:381-385
• recbole/model/sequential_recommender/sasreccpr.py:389-395

---

I have a PR ready with the fix and unit tests. Would you like me to submit it?

[rkawajiri]

Related Issue

This bug is also the root cause of Issue #1844.

Connection:

• Issue [🐛BUG] Discrepancies in Pop model metric values with different train_batch_size #1844 reports that Pop model metrics vary significantly with different
• Larger batch sizes show worse metrics (e.g., recall@10: 0.0721 → 0.0212 when batch_size increases from 1024 to 8192)
• The values are drastically different (e.g., item Config module #1: 106 → 24)

Root Cause:
Both issues stem from the same bug - duplicate items within a batch are only counted once. Larger batch sizes have more duplicates, so the undercount is more severe, leading to worse metrics.

Fix:
PR #2199 resolves both #2198 and #1844 by using scatter_add_() for proper duplicate item accumulation.