Handle model_max_length / max_length = 0 in tokenizer config (fixes #685)

[unusdon]

Fixes #685.

What changed

HuggingFace tokenizer configs may set model_max_length or max_length to 0 — used e.g. when a tokenizer stores its padding token inline in tokenizer.json without an external file. The prior key-existence check in load_tokenizer picked the 0 in the min(), then tokenizer.enable_truncation(max_length=0) silently disabled truncation, and any input past the underlying model's context window crashed downstream.

Switched to truthiness checks so 0 falls back to the other key (or to AssertionError when both are 0/missing).

Fix

     with open(str(tokenizer_config_path)) as tokenizer_config_file:
         tokenizer_config = json.load(tokenizer_config_file)
-        assert "model_max_length" in tokenizer_config or "max_length" in tokenizer_config, (
-            "Models without model_max_length or max_length are not supported."
+        model_max_length = tokenizer_config.get("model_max_length")
+        max_length = tokenizer_config.get("max_length")
+        assert model_max_length or max_length, (
+            "Models without a non-zero model_max_length or max_length are not supported."
         )
-        if "model_max_length" not in tokenizer_config:
-            max_context = tokenizer_config["max_length"]
-        elif "max_length" not in tokenizer_config:
-            max_context = tokenizer_config["model_max_length"]
+        if not model_max_length:
+            max_context = max_length
+        elif not max_length:
+            max_context = model_max_length
         else:
-            max_context = min(tokenizer_config["model_max_length"], tokenizer_config["max_length"])
+            max_context = min(model_max_length, max_length)

Handles both cases the reporter flagged (either key present as 0) as well as the missing-key cases the original code already covered.

Tests

Added tests/test_preprocessor_utils.py — 6 tests covering:

• model_max_length=0 falls back to max_length
• max_length=0 falls back to model_max_length
• both non-zero → min() still wins
• only one key present in the config (works as before)
• both zero → AssertionError (still fails loud)

All 6 pass locally against fastembed@0.8.0.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: debcc3d7-afb6-4875-b174-f4d836cd877e

📥 Commits

Reviewing files that changed from the base of the PR and between c48247f and 589c204.

📒 Files selected for processing (2)

• fastembed/common/preprocessor_utils.py
• tests/test_preprocessor_utils.py

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.

---

📝 Walkthrough

Walkthrough

load_tokenizer now treats missing, None, and zero tokenizer limits as uncapped values. It selects the available non-zero limit or the minimum of both positive limits, and raises an assertion when both limits are zero. New tests create minimal tokenizers and cover zero, missing, positive, and invalid limit configurations.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 589c2

The change makes zero-valued tokenizer limits fall back safely to a valid limit and adds coverage for the supported cases; no actionable merge-blocking risk remains after normal checks and review.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 12.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes handling zero values in tokenizer configuration and references the relevant issue.
Description check	✅ Passed	The description explains the bug, implementation change, expected behavior, and regression tests.
Linked Issues check	✅ Passed	The code and tests satisfy issue #685 by handling zero or missing values and preserving the minimum non-zero limit.
Out of Scope Changes check	✅ Passed	The changes are limited to tokenizer max-length handling and focused regression tests for issue #685.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}