[bugfix] AS block leaks#7895
Conversation
|
Thanks for your contribution! |
PaddlePaddle-bot
left a comment
PaddlePaddle-bot
left a comment
There was a problem hiding this comment.
🤖 Paddle-CI-Agent | pr_review |
2026-05-22 13:21:52
📋 Review 摘要
PR 概述：修复 resource_manager_v1 调度器中 AS block 泄漏问题，将 can_allocate_gpu_blocks 的阈值计算从内联表达式改为通过 _get_can_schedule_prefill_threshold_block() 函数获取，并对已知的 storage block 泄漏路径增加 Warning 注释
变更范围：fastdeploy/engine/sched/resource_manager_v1.py
影响面 Tag：[Scheduler]
问题
| 级别 | 文件 | 概述 |
|---|---|---|
| ❓ 疑问 | resource_manager_v1.py:1130 |
Warning 注释承认 _free_blocks 调用顺序会导致 storage block 泄漏，但代码仍执行该调用，未见实际修复 |
| 📝 PR 规范 | — | 标题 Tag 大小写不规范、release 分支缺 Cherry-Pick 标记、描述内容全部为空 |
📝 PR 规范检查
本 PR 存在以下规范问题：
- 标题 Tag 大小写偏差：
[bugfix]→ 官方 Tag 为[BugFix] - Cherry-Pick 格式缺失：目标分支为
release/2.6（非develop），标题应采用[Cherry-Pick][BugFix]格式并附原始 PR 编号 - 描述内容全部为空：Motivation / Modifications / Usage or Command / Accuracy Tests 均未填写
标题建议（可直接复制）：
[Cherry-Pick][BugFix] Fix AS block leaks in resource_manager_v1(#原PR号)
PR 描述建议（可直接复制）：
## Motivation
修复 resource_manager_v1 调度器中 AS block 泄漏问题。在分层 KV Cache（hierarchical cache）场景下，`can_allocate_gpu_blocks` 的阈值计算未考虑 AS block 需求，导致实际可用 block 不足时仍尝试分配，进而引发 block 泄漏及潜在死锁。
## Modifications
- `fastdeploy/engine/sched/resource_manager_v1.py`：将 `_allocate_decode_and_extend()` 中两处内联 block 数计算替换为 `_get_can_schedule_prefill_threshold_block()` 函数调用，以正确获取包含 AS block 的可调度阈值；对已知的 `_free_blocks` 调用顺序风险增加 Warning 注释
## Usage or Command
N/A
## Accuracy Tests
N/A
## Checklist
- [ ] Add at least a tag in the PR title.
- Tag list: [`[FDConfig]`,`[APIServer]`,`[Engine]`, `[Scheduler]`, `[PD Disaggregation]`, `[Executor]`, `[Graph Optimization]`, `[Speculative Decoding]`, `[RL]`, `[Models]`, `[Quantization]`, `[Loader]`, `[OP]`, `[KVCache]`, `[DataProcessor]`, `[BugFix]`, `[Docs]`, `[CI]`, `[Optimization]`, `[Feature]`, `[Benchmark]`, `[Others]`, `[XPU]`, `[HPU]`, `[GCU]`, `[DCU]`, `[Iluvatar]`, `[Metax]`]
- You can add new tags based on the PR content, but the semantics must be clear.
- [ ] Format your code, run `pre-commit` before commit.
- [ ] Add unit tests. Please write the reason in this PR if no unit tests.
- [ ] Provide accuracy results.
- [x] If the current PR is submitting to the `release` branch, make sure the PR has been submitted to the `develop` branch, then cherry-pick it to the `release` branch with the `[Cherry-Pick]` PR tag.总体评价
核心修复（两处对称地用 _get_can_schedule_prefill_threshold_block() 替代内联 block 数计算）思路正确，逻辑一致。但新增的 Warning 注释承认存在 storage block 泄漏路径（_free_blocks 先于 update_cache_blocks 执行），而 PR 标题正是 "AS block leaks"，需作者说明此泄漏路径是否已在本 PR 范围内修复，或有对应跟进计划；PR 规范信息亦需补全。
| self.req_dict[request.request_id] = allocated_position | ||
| llm_logger.debug(f"req_id:{request.request_id} allocate pos end") | ||
| else: | ||
| # Warning: _free_blocks before update_cache_blocks may cause storage blocks leak |
There was a problem hiding this comment.
❓ 疑问 新增 Warning 注释说明 _free_blocks 在 update_cache_blocks 之前调用可能导致 storage block 泄漏，但紧接着的代码仍然执行 self._free_blocks(request)，并未修复该顺序问题。
PR 标题为 "AS block leaks"，请作者说明：
- 此处
_free_blocksbeforeupdate_cache_blocks的调用顺序是否就是本次要修复的 AS block 泄漏根因？ - 若是，应在此处补充修复逻辑（例如先调用
update_cache_blocks再调用_free_blocks，或跳过_free_blocks）；若不是，建议在 PR 描述中补充对应 issue 编号，说明此 Warning 何时跟进修复。
CI报告基于以下代码生成（30分钟更新一次）: 1 任务总览当前 Required 任务 9/10 通过，唯一阻塞项为
2 任务状态汇总日志列说明：失败任务直接使用日志链接；运行中任务使用 Job 链接。 2.1 Required任务 : 9/10 通过
2.2 可选任务 — 22/26 通过
3 失败详情（仅 required）Approval — 等待人工审批（置信度: 高）该 Job 需要人工 Approval，完成审批后 CI 才会继续执行。
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## release/2.6 #7895 +/- ##
==============================================
Coverage ? 72.34%
==============================================
Files ? 381
Lines ? 54227
Branches ? 8473
==============================================
Hits ? 39233
Misses ? 12226
Partials ? 2768
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
3 participants
Motivation
Modifications
Usage or Command
Accuracy Tests
Checklist
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.