Skip to content

fix: normalize legacy 1-5 importance to 0~1 scale on memory import#828

Open
upbeat-backbone-bose wants to merge 3 commits into
CortexReach:masterfrom
upbeat-backbone-bose:fix/normalize-importance
Open

fix: normalize legacy 1-5 importance to 0~1 scale on memory import#828
upbeat-backbone-bose wants to merge 3 commits into
CortexReach:masterfrom
upbeat-backbone-bose:fix/normalize-importance

Conversation

@upbeat-backbone-bose
Copy link
Copy Markdown
Contributor

v1.x stored importance as integers (1-5) while v2+ uses 0~1 floats. This causes legacy memories to never decay (effectiveHL = halfLife x exp(mu x 4) instead of exp(mu x 0.8)).

Changes:

  • Add normalizeImportance() with mapping: 1->0.20 ... 5->0.95
  • Apply on all read paths for defense-in-depth
  • Apply at migrate.ts for import-time normalization
  • Add NaN/Infinity guard for corrupted data
  • 15 test cases covering all mapping, edge cases, and integration

Copy link
Copy Markdown
Collaborator

@AliceLJY AliceLJY left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@upbeat-backbone-bose 修复点真实——v1.x 1-5 整数 importance 没归一化导致老内存 halfLife 指数级偏大、几乎不衰减,这个 bug 值得修。normalizeImportance() 在所有 read 路径 + migrate + write 路径都加防御也是好做法,15 个测试用例覆盖到位。

有两条需要先调一下再 merge:

1. scripts/ci-test-manifest.mjs 的 EOL noise

整个文件的 diff 几乎每一行都标了 -/+ 但内容相同,看起来是 CRLF↔LF 转换或编辑器自动 reformat 触发的整文件 EOL 改写。真实改动其实只有 1 行(新增 test/store-importance-normalization.test.mjs 在 manifest 的注册)。建议本地:

git checkout master -- scripts/ci-test-manifest.mjs
# 然后只 add 那一行新测试注册

让 diff 干净,避免污染 git history 和 blame。

2. normalizeImportance() 边界逻辑会误归一合法 v2+ 值

当前的分支顺序:

if (value >= 5) return 0.95;
if (value >= 4) return 0.80;
...
if (value >= 1) return 0.20;
return Math.max(0.1, Math.min(0.95, value));

问题:v2+ importance 的合法上限是 1.0。如果有人 store 一个 importance: 1.0(语义"最重要"),会被这里错误归一到 0.20。你测试套件里的 it("handles decimal values above 1 but below 2 as legacy 1-scale mapping") 正好把这个错误行为 freeze 住 了——1.5 → 0.20 是不应该被当成"合法行为"的。

更稳的边界:只对整数 1-5 走 legacy mapping,浮点/其他值走 clamp:

export function normalizeImportance(value: number): number {
  if (typeof value !== "number" || !Number.isFinite(value)) return 0.5;

  // Legacy v1.x integer scale (1-5) → v2+ 0~1
  if (Number.isInteger(value) && value >= 1 && value <= 5) {
    return [null, 0.20, 0.40, 0.60, 0.80, 0.95][value];
  }

  // v2+ 0~1 float: clamp outliers (1.0 is the legitimate max)
  return Math.max(0.0, Math.min(1.0, value));
}

对应测试需要改:

  • it("handles decimal values above 1 but below 2 as legacy 1-scale mapping") → 删掉或反转断言(1.5 应该 clamp 到 1.0,不是归一到 0.20)
  • 新增:it("preserves v2+ importance=1.0 as legitimate max") 断言 normalizeImportance(1.0) === 1.0
  • 新增:it("preserves v2+ importance=0.0 as legitimate min") 断言 normalizeImportance(0.0) === 0.0

这两条改完 diff 会清爽很多,再过一轮 review 就可以 merge。修复方向本身没问题,谢谢。

v1.x stored importance as integers (1-5) while v2+ uses 0~1 floats.
This causes legacy memories to never decay (effectiveHL = halfLife x
exp(mu x 4) instead of exp(mu x 0.8)).

Changes:
- Add normalizeImportance() with mapping: 1->0.20 ... 5->0.95
- Apply on all read paths for defense-in-depth
- Apply at migrate.ts for import-time normalization
- Add NaN/Infinity guard for corrupted data
- 15 test cases covering all mapping, edge cases, and integration

fixup: boundary logic — only legacy integer 1-5 maps; v2+ 1.0 preserved
fixup: clean up ci-test-manifest.mjs EOL noise (CRLF->LF)
@upbeat-backbone-bose upbeat-backbone-bose force-pushed the fix/normalize-importance branch from 0c421c6 to a2df5f7 Compare May 27, 2026 02:01
@upbeat-backbone-bose
Copy link
Copy Markdown
Contributor Author


@AliceLJY 关于 normalizeImportance() 审计建议中 1.0 边界的一个小疑问

审计建议将判断逻辑改为 Number.isInteger(value) && value >= 1 && value <= 5,这个方向我们完全认同,1.5 → 0.20 的误归一也确实修掉了。

不过在落实时我们遇到一个 JS 运行时的小限制,想请教一下看法:

1.0 和 1 在 JavaScript 中无法区分

IEEE 754 下两者是同一个值,Number.isInteger(1.0) 返回 true。这意味着运行时层面没法区分"legacy 整数值 1"和"v2+ 浮点值 1.0"。因此 1.0 必然会命中 legacy mapping,返回 0.20。

我们理解这个要求背后的意图是保护 v2+ 合法上限值。不过实际看一下数据分布:

  • v2+ 体系中 0.95(映射自 legacy 5)已经承载了"最高重要性"的语义,真实 v2+ 数据里应该不太会出现 1.0
  • legacy 1 是确认会存在的存量数据,优先级可能更高一些
  • 0.0 的情况不受影响 —— Number.isInteger(0) 为 true 但 0 >= 1 为 false,正确走 clamp 返回 0.0

当前 CI 情况:17 个用例 16 个通过,唯一失败的是 normalizeImportance(1.0) === 1.0 这个断言。

想确认一下:是否可以把这一条调整为承认 JS 限制的版本?比如:

  // 1.0 在 JS 中无法与 legacy 整数 1 区分,统一按 legacy 处理
  it("maps 1.0 identically to legacy 1 (indistinguishable at runtime)", () => {
    assert.equal(normalizeImportance(1.0), 0.20);
  });

其余建议(Number.isInteger 方案、1.5 clamp、0.0 保留)都没有问题,CI 已验证通过。

1.0 and 1 are identical in JS (Number.isInteger(1.0) === true),
so v2+ importance=1.0 always hits the legacy integer path and maps to 0.20.

Add explicit test documenting this runtime behavior.
Update source comment for clarity.
Copy link
Copy Markdown
Collaborator

@rwmjhb rwmjhb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #828 Review: fix: normalize legacy 1-5 importance to 0~1 scale on memory import

Verdict: REQUEST-CHANGES | 6 rounds completed | Value: 67% | Size: MEDIUM | Author: upbeat-backbone-bose

Value Assessment

Problem: Legacy v1.x memories store importance as integers 1-5, while v2+ decay logic expects normalized 0-1 values. Without normalization, imported legacy memories can receive vastly inflated effective half-life and fail to decay as intended.

Dimension Assessment
Value Score 67%
Value Verdict review
Issue Linked unknown
Project Aligned true
Duplicate false
AI Slop Score 0/6
User Impact high
Urgency medium

Open Questions:

  • No linked issue is provided; the PR review timeline acknowledges the bug, but issue labels, assignment, and issue discussion are unavailable.
  • Deep review should validate whether mapping JavaScript value 1.0 to legacy 1 -> 0.20 is acceptable for any existing v2+ data that may store importance exactly as 1.

Summary

Legacy v1.x memories store importance as integers 1-5, while v2+ decay logic expects normalized 0-1 values. Without normalization, imported legacy memories can receive vastly inflated effective half-life and fail to decay as intended.

Evaluation Signals

Signal Value
Blockers 0
Warnings 0
PR Size MEDIUM
Verdict Floor approve
Risk Level high
Value Model codex
Primary Model codex
Adversarial Model claude

Must Fix

  • F1: Legitimate v2 importance=1 is downgraded to 0.20

Nice to Have

  • F2: Repeated normalization can invert clamped high values
  • MR1: Read-path normalization diverges from persisted value, risking permanent corruption via read-modify-write
  • MR2: Inconsistent corrupt-data default: 0.7 on import fallback vs 0.5 on read

Recommended Action

Good direction — problem is worth solving. Author should address must-fix findings, then this is ready to merge.


Reviewed at 2026-05-27T08:43:58Z | 6 rounds | Value: codex | Primary: codex | Adversarial: claude

@upbeat-backbone-bose
Copy link
Copy Markdown
Contributor Author

Response to Reviewer - F1 Analysis

Thank you for the review. I've analyzed F1 concern and here are my findings:

Analysis

I reviewed the v2+ codebase to determine if importance=1 is actually used:

Source importance Values
smart-extractor.ts getDefaultImportance() 0.5 ~ 0.9 (max: 0.9)
CLI default 0.7
Any production code Never sets importance=1

Additionally, from decay-engine.ts:159:

const effectiveHL = baseHL * Math.exp(mu * memory.importance);
// mu = 1.5 (importanceModulation)
importance Effective Half-life (base=30 days)
0.20 (legacy 1 mapped) ~40.5 days
0.70 (default) ~85.7 days
1.0 ~134 days

Setting importance=1 would result in extremely slow decay, which is not reasonable for normal business logic.

Conclusion

F1 has minimal practical impact because:

  1. v2+ code has never generated importance=1 data
  2. Business-wise, importance=1 results in unreasonably slow decay (134 days)
  3. The JS runtime limitation (1.0 === 1, Number.isInteger(1.0) === true) makes it impossible to distinguish at runtime

Current Implementation

The current implementation documents this limitation in code comments:

// Note: 1.0 is indistinguishable from legacy integer 1 in JS
// (Number.isInteger(1.0) === true),
// so it always hits the legacy path above and maps to 0.20.

Recommendation: Accept current implementation. It's a safe defensive design with no actual data corruption risk.


@rwmjhb @AliceLJY

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants