Skip to content

fix(api): return 429 when server is busy - #1441

Merged
sufubao merged 1 commit into
ModelTC:mainfrom
sufubao:fix/server-busy-429
Aug 6, 2026
Merged

fix(api): return 429 when server is busy#1441
sufubao merged 1 commit into
ModelTC:mainfrom
sufubao:fix/server-busy-429

Conversation

@sufubao

@sufubao sufubao commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • change the default ServerBusyError status from 503 to 429 and label OpenAI-style responses as RateLimitError
  • prime streaming responses only in PD-master mode before Starlette sends response headers, so busy failures before the first generated item return HTTP 429 without changing normal-mode header/disconnect behavior
  • treat a timeout after streaming has started as an in-flight server_error SSE event, not as a 429/rate-limit rejection
  • handle server-busy failures from /get_score
  • preserve the Anthropic rate_limit_error envelope for /v1/messages before streaming starts
  • log expected pre-stream server-busy control flow at warning level without a traceback

Checks

  • python -m pytest -q test/test_api/test_anthropic_extra_body.py (36 passed)
  • PD-only stream priming behavior check: normal mode consumes zero items; PD-master replays the prefetched item and propagates a pre-first-item ServerBusyError
  • pre/post stream classification check: pre-stream busy propagates for HTTP 429; post-start timeout emits server_error / stream_error
  • pre-commit run --files lightllm/server/api_anthropic.py lightllm/server/api_http.py lightllm/server/api_openai.py lightllm/server/api_responses.py lightllm/utils/error_utils.py
  • python -m py_compile lightllm/server/api_anthropic.py lightllm/server/api_http.py lightllm/server/api_openai.py lightllm/server/api_responses.py lightllm/utils/error_utils.py

@sufubao
sufubao force-pushed the fix/server-busy-429 branch 3 times, most recently from 885aff2 to 8187a7d Compare August 6, 2026 12:30
@shihaobai
shihaobai self-requested a review August 6, 2026 12:42
@sufubao
sufubao force-pushed the fix/server-busy-429 branch 2 times, most recently from 5aa3286 to 847bb3c Compare August 6, 2026 13:24
@sufubao
sufubao force-pushed the fix/server-busy-429 branch from 847bb3c to 847aff9 Compare August 6, 2026 13:31
@sufubao
sufubao merged commit 8f5ac4f into ModelTC:main Aug 6, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant