[Models] add fleet model fallback#7732
Conversation
xiaoguoguo626807
had a problem deploying
to
Metax_ci
GitHub Actions
Failure
|
Thanks for your contribution! |
xiaoguoguo626807
had a problem deploying
to
Metax_ci
GitHub Actions
Failure
xiaoguoguo626807
had a problem deploying
to
Metax_ci
GitHub Actions
Failure
CI报告基于以下代码生成（30分钟更新一次）: 1 任务总览Required 任务当前 8/10 通过，仍有 2 个 required 失败任务 需要处理：
2 任务状态汇总日志列说明：失败任务直接使用 CI 日志链接；Required 失败任务已在表格中给出根因和修复建议。 2.1 Required任务 : 8/10 通过
2.2 可选任务 — 29/32 通过
3 失败详情（仅 required）Run FastDeploy Unit Tests and Coverage / run_tests_with_coverage — 覆盖率门禁（置信度: 高）Run FastDeploy Unit Tests and Coverage / run_tests_with_coverage
失败用例: 无。日志显示 根因详情: 关键日志: 修复建议:
修复建议摘要: 补充 base_fleet 相关单测 关联变更: Approval — 需要人工审批（置信度: 高）该 Job 需要人工 Approval，完成审批后 CI 才会继续执行。 |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #7732 +/- ##
==========================================
Coverage ? 63.65%
==========================================
Files ? 468
Lines ? 65247
Branches ? 10015
==========================================
Hits ? 41536
Misses ? 20886
Partials ? 2825
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
xiaoguoguo626807
had a problem deploying
to
Metax_ci
GitHub Actions
Failure
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
xiaoguoguo626807
had a problem deploying
to
Metax_ci
GitHub Actions
Failure
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
PaddlePaddle-bot
left a comment
PaddlePaddle-bot
left a comment
There was a problem hiding this comment.
🤖 Paddle-CI-Agent | pr_review |
2026-05-25 14:54:00
📋 Review 摘要
PR 概述：新增 PaddleFleet 作为模型推理后端，通过替换 PaddleFleet TransformerLayer 中的 core_attention 为 FastDeploy Attention 内核，复用 FastDeploy 的 KV Cache 和高性能 Attention 计算。
变更范围：model_executor/models/、config.py、engine/args_utils.py、graph_optimization/decorator.py
影响面 Tag：[Models] [FDConfig] [Engine]
问题
| 级别 | 文件 | 概述 |
|---|---|---|
| 🔴 Bug | model_base.py:203 |
pip 安装命令字符串拼接缺少空格，命令无法执行 |
| 🟡 建议 | base_fleet.py |
assert 用于运行时校验，Python -O 优化模式下会被跳过 |
| 🟡 建议 | base_fleet.py |
load_weights 为空实现（pass），存在框架调用被静默跳过的风险 |
| ❓ 疑问 | base_fleet.py |
fd_layer_id = layer_number：PaddleFleet layer_number 从 1 开始，是否与 FastDeploy 0-indexed layer_id 对齐？ |
| 📝 PR 规范 | — | Checklist [x] Provide accuracy results 与 Accuracy Tests: N/A 不一致 |
🟡 base_fleet.py — assert 用于运行时校验
FastDeployAttention.forward 中两处 assert 用于防御性输入校验，在 Python -O 模式下会被静默跳过：
# 当前（有风险）
assert forward_meta is not None, "forward_meta must be provided"
assert kv_compressed is not None, "kv_compressed must be provided when use"
# 建议改为
if forward_meta is None:
raise ValueError("forward_meta must be provided in config")
if kv_compressed is None:
raise ValueError("kv_compressed must be provided when multi_latent_attention is enabled")🟡 base_fleet.py — load_weights 空实现
load_weights 方法为 pass，但框架在动态加载或热更新场景可能调用此方法，导致权重被静默跳过。权重已在 __init__ 中通过 AutoModelForCausalLM.from_pretrained 加载，建议明确记录此行为或在被框架调用时保护：
@paddle.no_grad()
def load_weights(self, weights: Iterable[tuple[str, paddle.Tensor]]):
# Weights are loaded in __init__ via AutoModelForCausalLM.from_pretrained.
# This method is intentionally a no-op for PaddleFleet backend.
pass至少添加注释说明，避免后续维护者误认为这是遗漏实现。
❓ base_fleet.py — layer_id 1-indexed vs 0-indexed 对齐
在 patch_paddlefleet_core_attention 中：
fd_layer_id = layer_number # PaddleFleet layer_number 从 1 开始
fd_attn_instance = Attention(fd_config=fd_config, layer_id=fd_layer_id)PaddleFleet 的 layer_number 从 1 开始，FastDeploy 的 KV Cache 通常使用 0-indexed layer_id。如果 Attention 的 layer_id 是 0-indexed，则第 0 层 KV Cache slot 永远不会被使用，且最后一层会越界。
建议确认 Attention 类的 layer_id 语义，如果是 0-indexed 则应改为：
fd_layer_id = layer_number - 1📝 PR 规范检查
Checklist 中 [x] Provide accuracy results. 被勾选，但 ## Accuracy Tests 节内容为 N/A，两者不一致。对于新引入的推理后端，建议补充与参考实现的 logits 对齐结果，或明确在 Checklist 中取消勾选并在 Accuracy Tests 段说明原因。
PR 描述建议（可直接复制，仅修改 Checklist 勾选状态以修正不一致）：
## Motivation
新增 PaddleFleet 作为模型推理后端（`--model-impl paddlefleet`），通过将 PaddleFleet TransformerLayer 中的 `core_attention` 替换为 FastDeploy Attention 内核，实现在 PaddleFleet 模型结构上复用 FastDeploy 的 KV Cache 和高性能 Attention 计算。
## Modifications
- `config.py`: 新增 `paddlefleet` 到 `ModelImpl` 类型定义
- `engine/args_utils.py`: 支持 `--model-impl paddlefleet` CLI 参数，并补充校验逻辑
- `model_executor/models/paddleformers/base_fleet.py`: 新增 `PaddleFleetModelBase` 基类、`FastDeployAttention` 层及 `patch_paddlefleet_core_attention` 替换函数
- `model_executor/models/paddleformers/__init__.py`: 注册 `PaddleFleetForCausalLM` 模型类
## Usage or Command
```bash
python -m fastdeploy.entrypoints.openai.api_server \
--model /path/to/model \
--model-impl paddlefleet
```
## Accuracy Tests
N/A（本 PR 新增 PaddleFleet 推理后端，尚未提供与参考实现的 logits 对齐数据）
## Checklist
- [x] Add at least a tag in the PR title.
- Tag list: [`[FDConfig]`,`[APIServer]`,`[Engine]`, `[Scheduler]`, `[PD Disaggregation]`, `[Executor]`, `[Graph Optimization]`, `[Speculative Decoding]`, `[RL]`, `[Models]`, `[Quantization]`, `[Loader]`, `[OP]`, `[KVCache]`, `[DataProcessor]`, `[BugFix]`, `[Docs]`, `[CI]`, `[Optimization]`, `[Feature]`, `[Benchmark]`, `[Others]`, `[XPU]`, `[HPU]`, `[GCU]`, `[DCU]`, `[Iluvatar]`, `[Metax]`]
- You can add new tags based on the PR content, but the semantics must be clear.
- [x] Format your code, run `pre-commit` before commit.
- [ ] Add unit tests. Please write the reason in this PR if no unit tests.
- [ ] Provide accuracy results.
- [ ] If the current PR is submitting to the `release` branch, make sure the PR has been submitted to the `develop` branch, then cherry-pick it to the `release` branch with the `[Cherry-Pick]` PR tag.总体评价
整体实现思路清晰，通过 core_attention 替换实现了 PaddleFleet 与 FastDeploy 的对接。存在一处 🔴 Bug（pip 命令字符串拼接缺少空格），需修复后才能正确指引用户安装依赖；另有 layer_id 1-indexed 对齐问题值得作者确认，以避免潜在 KV Cache 越界。
| if is_paddlefleet_available(): | ||
| backend_arch = "PaddleFleetForCausalLM" | ||
| else: | ||
| raise ImportError( |
There was a problem hiding this comment.
🔴 Bug pip 安装命令字符串拼接缺少空格，命令无法执行
第3行字符串末尾缺少空格，与第4行直接拼接后变为 ...dev20260507--extra-index-url...，用户复制执行会报错。
建议修复：
"python -m pip install paddlefleet==0.3.0.dev20260507 "
"--extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/ "
"--extra-index-url https://www.paddlepaddle.org.cn/packages/nightly/cu126/"即在 dev20260507 后面加一个空格。
xiaoguoguo626807
had a problem deploying
to
Metax_ci
GitHub Actions
Failure
3 participants
Motivation
新增 PaddleFleet 作为模型推理后端（
--model-impl paddlefleet），通过将 PaddleFleet TransformerLayer 中的core_attention替换为 FastDeploy Attention 内核，实现在 PaddleFleet 模型结构上复用 FastDeploy 的 KV Cache 和高性能 Attention 计算。Modifications
config.py: 新增paddlefleet到ModelImpl类型定义engine/args_utils.py: 支持--model-impl paddlefleetCLI 参数，并补充校验逻辑model_executor/models/paddleformers/base_fleet.py: 新增PaddleFleetModelBase基类、FastDeployAttention层及patch_paddlefleet_core_attention替换函数model_executor/models/paddleformers/__init__.py: 注册PaddleFleetForCausalLM模型类Usage or Command
python -m fastdeploy.entrypoints.openai.api_server \ --model /path/to/model \ --model-impl paddlefleetAccuracy Tests
N/A（本 PR 新增 PaddleFleet 推理后端，尚未提供与参考实现的 logits 对齐数据）
Checklist
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.