Fix duplicate detection: lower default threshold and log near-misses

[nmbrthirteen]

Problem

Dedup was silently failing on the live site: multiple near-identical posts (e.g. four "Entain exits Poland/Croatia" articles) published as separate stories.

Root cause

On OpenAI (text-embedding-3-small), the pipeline similarity_threshold default of 0.82 is too high. Real cross-source rewrites of the same story score ~0.60-0.80, so Clusterer::match() almost never cleared the threshold and every article started a new cluster.

Worse, the score < threshold branch returned null with no log, so the Activity log showed nothing (the existing "Close call" line only fires when score ≥ threshold but facts don't overlap). The failure was invisible.

Changes

• Lower default similarity_threshold 0.82 → 0.72 (Settings.php, Admin.php)
• Log the nearest-story score when it falls below the threshold, so the miscalibration is visible and tunable from the Activity log (Clusterer.php)
• Add help text to the "Duplicate sensitivity" setting explaining direction and default

The shared-salient-fact guard (signal 3) still blocks false merges, so a lower embedding threshold is safe.

Note for existing installs

Lowering the code default does not change a stored option. Existing sites must set Duplicate sensitivity → 0.72 in Settings for the fix to take effect.

Testing

86 tests pass, lint clean.

Summary by CodeRabbit

• Bug Fixes
  • Updated the default duplicate sensitivity value to 0.72 when no setting is provided, making duplicate detection less strict by default.
  • Improved logging when an item is treated as a new story instead of a duplicate, helping with troubleshooting and review.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8fc7ce2f-70e9-4b2c-9030-142fc0d68675

📥 Commits

Reviewing files that changed from the base of the PR and between 973f5c5 and 22b0bb5.

📒 Files selected for processing (4)

• src/Admin/Admin.php
• src/Admin/views/settings.php
• src/Cluster/Clusterer.php
• src/Settings.php

---

📝 Walkthrough

Walkthrough

The default similarity threshold fallback value was lowered from 0.82 to 0.72 in both Settings and Admin classes. The settings view markup was reformatted. Clusterer now logs an info-level event when a candidate's similarity score falls below the threshold, treating the item as a new story.

Changes

Similarity Threshold Update

Layer / File(s)	Summary
Default threshold value change src/Settings.php, src/Admin/Admin.php, src/Admin/views/settings.php	Fallback default for similarity_threshold changed from 0.82 to 0.72 in the getter and save handler; settings view markup reformatted without content changes.
Below-threshold duplicate logging src/Cluster/Clusterer.php	Adds an EventLog::info() call logging candidate story ID, score, and threshold when a match falls below threshold and is treated as a new story.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Related issues: None identified from the provided information.

Related PRs: None identified from the provided information.

Suggested labels: enhancement, settings, logging

Suggested reviewers: None identified from the provided information.

🌱 Poem

A whisker tighter, threshold shrinks,
0.82 down to 0.72, methinks.
Duplicates now confess in logs,
While settings forms shed their clogs.
A rabbit hops through cleaner code — hooray! 🐇

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: lowering the duplicate detection threshold and adding near-miss logging.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/dedup-threshold-calibration

Warning

Tools execution failed with the following error:

Failed to run tools: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error)

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}