Skip to content

Fix duplicate RSS items when pubDate is missing#743

Closed
kion wants to merge 2 commits intoyang991178:masterfrom
kion:fix-feed-duplicate-items
Closed

Fix duplicate RSS items when pubDate is missing#743
kion wants to merge 2 commits intoyang991178:masterfrom
kion:fix-feed-duplicate-items

Conversation

@kion
Copy link
Copy Markdown

@kion kion commented Nov 5, 2025

Issue

RSS feed items without a pubDate field are duplicated on every feed refresh.

Example feed: https://rsshub.app/anthropic/engineering
The article "Writing effective tools for agents — with agents" lacks a pubDate, causing it to appear at the top of the feed list after each refresh.

fluent-reader-duplicate-items

Root Cause

When an RSS item has no pubDate or isoDate, the RSSItem constructor falls back to the current fetch time:

this.date = new Date(item.isoDate ?? item.pubDate ?? this.fetchedDate)

The original deduplication logic used (source, title, date) as a composite key. Since the date changes on each fetch, the duplicate check fails and the same item gets inserted repeatedly.

Solution

Use the item's link as the primary deduplication key instead of the date-based composite key:

  • If link is present (vast majority of feeds): match on (source, link)
  • If link is missing (rare edge case): fall back to (source, title, date)

Also fixed sorting for items without pubDate. Previously, these items appeared at the top of the feed (sorted as "newest" since they used the current fetch time). Now they appear at the bottom while preserving their feed order.

Testing

Tested with multiple RSS feeds, including the problematic aforementioned one, as well as multiple others that worked fine initially (to confirm nothing got broken). No duplicates appear after multiple refreshes. Items without publication date appear on the bottom of the list.

@kion kion closed this Nov 8, 2025
@kion kion deleted the fix-feed-duplicate-items branch November 8, 2025 20:55
@kion kion restored the fix-feed-duplicate-items branch November 8, 2025 21:08
@kion kion reopened this Nov 8, 2025
@kion kion closed this Mar 14, 2026
@kion kion deleted the fix-feed-duplicate-items branch March 14, 2026 02:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant