fix: ensure gateway restarts on install failure via ERR trap#2152
fix: ensure gateway restarts on install failure via ERR trap#2152Zhe-SH-CN wants to merge 21 commits into
Conversation
* feat(api): align OpenMem v1 SDK with cloud API
Co-authored-by: shinetata <149466187+shinetata@users.noreply.github.com>
…#2113) Rewrite scheduler Quick Start examples to call the real /product/scheduler REST endpoints with requests instead of non-existent MemOSClient methods; align response fields with the actual handler output; rename the mis-named ' wait.md' to 'wait.md'. Closes MemTensor#2083 Co-authored-by: sunqi <sunqi@memtensor.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…ensor#2112) fix(api): show structured add example in /docs (messages not string) Swagger UI rendered APIADDRequest.messages as "string" because it picks the leading `str` branch of the `str | MessageList | RawMessageList` union when generating the example. Add a model-level json_schema_extra example so /docs shows a copy-paste-ready payload with a structured messages list. Schema-only change: field types, validation and runtime behaviour are unchanged, so the core add pipeline is unaffected. Closes MemTensor#1505 Co-authored-by: sunqi <sunqi@memtensor.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…emTensor#2096) The hermes adapter's __init__.py imported MemosHttpClient from bridge_client, but the class was never committed — every plugin python test failed at collection and any real Hermes host loading memos_provider hit ImportError at module load. The phantom usages arrived via the pr/may27-fixes merge (0398e0e) as a half-committed HTTP-bridge feature. Revert the four usages and the HTTP-first branches in initialize / _reconnect_bridge, remove the _connect_http_bridge helper and the now- unused probe_viewer_status / startup_lock_active imports (their definitions in daemon_manager.py are untouched for a future HTTP PR). Add test_module_imports_cleanly regression test asserting MemTensorProvider / MemosBridgeClient / BridgeError are present on the real contract surfaces and MemosHttpClient is absent on both memos_provider and bridge_client, so this class of import-level breakage cannot silently return. Restores the adapter to stdio-only behavior. Net diff: 57 additions / 54 deletions across the adapter plus the regression test. Fixes MemTensor#2096
Avoid re-running dictConfig on every get_logger call, since repeated configuration can close and replace active handlers while background threads are emitting records. Keep logging configuration process-local and reconfigure only after a PID change.
Use FastAPI lifespan cleanup to stop scheduler and RabbitMQ resources before logging handlers are torn down during worker or pod shutdown.
## Description fix: configure logging once per process fix: close scheduler resources on API shutdown ## Type of change - [x] Bug fix (non-breaking change which fixes an issue) ## How Has This Been Tested? - [x] Unit Test - tests/api/test_lifecycle.py, tests/test_log.py ## Checklist - [x] I have performed a self-review of my own code | 我已自行检查了自己的代码 - [x] I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释 - [x] I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常 - [ ] I have created related documentation issue/PR in [MemOS-Docs](https://github.com/MemTensor/MemOS-Docs) (if applicable) | 我已在 [MemOS-Docs](https://github.com/MemTensor/MemOS-Docs) 中创建了相关的文档 issue/PR(如果适用) - [x] I have linked the issue to this PR (if applicable) | 我已将 issue 链接到此 PR(如果适用) - [x] I have mentioned the person who will review this PR | 我已提及将审查此 PR 的人 ## Reviewer Checklist - [ ] closes #xxxx (Replace xxxx with the GitHub issue number) - [ ] Made sure Checks passed - [ ] Tests have been provided
* Fix MemTensor#1540: fix: (MemTensor#1824) docs(memos-local-plugin): clarify install path and stale dir names (MemTensor#1540) The README's 'Quick start' section told users to use install.sh instead of npm install, but the warning was buried and users still tried 'npm install -g @memtensor/memos-local-plugin' first. The reporter in MemTensor#1540 encountered this on a Hermes deployment. This change: - Promotes the 'do not run npm install -g' notice to a prominent IMPORTANT callout explaining why global install is wrong (no agent-home deploy, no config.yaml, no bridge/viewer) and that the tarball intentionally ships built artifacts only. - Adds a Troubleshooting subsection covering the two specific symptoms in the bug report: the 'package not found' misread, and the stale web/ and site/ directory names (web/ is now viewer/, site/ was removed by commit 26e7e3d). - Mentions install.ps1 for Windows alongside install.sh. - CHANGELOG: record the docs fix and reference MemTensor#1540. Documentation-only change; no code or runtime behavior touched. Co-authored-by: MemOS AutoDev <autodev@memtensor.ai> Co-authored-by: Matthew <heimixiaozhuang@zju.edu.cn> * Fix MemTensor#1888: [Bug] test_system_parser.py: SystemParser.__init__() got an unexpected keyword a (MemTensor#1889) fix: remove invalid chunker parameter from SystemParser test instantiation - SystemParser.__init__() signature changed to (embedder, llm=None) - Test was still passing chunker=None causing TypeError - Fixes all 5 failing tests in test_system_parser.py Fixes MemTensor#1888 Co-authored-by: MemOS AutoDev <autodev@memos.ai> Co-authored-by: Matthew <heimixiaozhuang@zju.edu.cn> * Fix MemTensor#1525: [Bug] clean_json_response crashes with cryptic AttributeError when given None (MemTensor#1884) * test: add comprehensive tests for clean_json_response (issue MemTensor#1525) - Add test suite in tests/mem_os/test_format_utils.py - Cover None input ValueError with diagnostic message - Cover markdown removal, whitespace stripping, edge cases - Verify fix for AttributeError when LLM returns None * style: format clean_json_response tests --------- Co-authored-by: MemOS AutoDev <autodev@memos.ai> Co-authored-by: Matthew <heimixiaozhuang@zju.edu.cn> * Fix MemTensor#1901: share_cube_with_user passes swapped args to _validate_cube_access — fails for ev (MemTensor#1903) fix: validate current user not target in share_cube_with_user (MemTensor#1901) share_cube_with_user(cube_id, target_user_id) called _validate_cube_access(cube_id, target_user_id), but the validator signature is (user_id, cube_id). The cube_id therefore landed in the user_id slot and _validate_user_exists raised "User '<cube_id>' does not exist or is inactive" for every well-formed call, making the API unusable. The in-code comment "Validate current user has access to this cube" already documented the correct intent: the sharing user (self.user_id) must have access to the cube being shared, not the target. Switch the call to self._validate_cube_access(self.user_id, cube_id). The target user's existence is independently checked on the next line via validate_user(target_user_id), so that path is unchanged. Add regression tests in tests/mem_os/test_memos_core.py that pin down: - validate_user_cube_access is consulted with (self.user_id, cube_id), - add_user_to_cube is called with (target_user_id, cube_id) on success, - a missing target raises "Target user '<id>' does not exist". Closes MemTensor#1901 Co-authored-by: MemOS AutoDev Bot <autodev@memtensor.local> Co-authored-by: Matthew <heimixiaozhuang@zju.edu.cn> * feat(memos): chunk batch reflection scoring * Fix MemTensor#1897: bug(memos-local-plugin): Hermes background model status checks can trigger paid (MemTensor#1899) * Fix MemTensor#1897: fix(memos-local-plugin): add LLM circuit breaker for terminal provider errors Issue MemTensor#1897 reported ~12,900 paid LLM requests in 24 h on Hermes against a DeepSeek key with insufficient balance. The local `system_model_status` row count (12,900) closely tracked the provider-side `request_count` (11,344) for the same billing window. The naming is misleading: `system_model_status` is not a health probe; it is the audit row written once per LLM call (ok / fallback / error) inside `core/llm/client.ts`. With no circuit breaker, every pipeline subscriber (capture / session-relation / reward / L2 / L3 / skill / retrieval LLM filter / world-model) kept firing on every turn / closed episode / induction, generating one paid request each. Add a per-`LlmClient` circuit breaker: - Trips on terminal errors: HTTP 401/402/403 or messages containing `insufficient balance` / `invalid api key` / `unauthorized` / `account suspended` / `billing`. - Open: short-circuits subsequent calls inside the facade without contacting the provider. Throws `MemosError(LLM_UNAVAILABLE)` with `details.circuitOpen=true` so existing catch blocks still work. - Half-open after cool-down (default 5 min, configurable, min 30 s): next call probes the provider; success closes the breaker, terminal failure re-opens it for another cool-down. - Host fallback rescues a call without tripping the breaker — fallback exists precisely to keep going when the primary is down. - Coalesces `system_model_status="circuit_open"` audit rows to at most one per ~25 s while the breaker stays open, so we don't replace paid spam with audit-row spam. - Exposes `circuitOpen` / `circuitOpenUntil` / `circuitOpenedReason` via `LlmClientStats` for the Overview viewer card. - Enabled by default; legacy behaviour available via `circuitBreaker.enabled = false`. Tests: 9 new vitest cases under `tests/unit/llm/client.test.ts` covering trip on 402, trip on "insufficient balance" message, no trip on generic transient, coalescing, half-open close on success, host-fallback rescues without trip, disabled mode, stats fields, and re-open on terminal probe failure. All 59 LLM and 28 pipeline tests pass; `tsc --noEmit` clean. Out of scope (tracked separately): 429 `Retry-After` handling (issue MemTensor#1620), per-tool rate limits, daily budget caps. * Fix LLM breaker with host fallback --------- Co-authored-by: autodev-bot <autodev@memtensor.local> Co-authored-by: Jiang <33757498+hijzy@users.noreply.github.com> Co-authored-by: GU TIANCHUN <96930846+TianchunGu@users.noreply.github.com> Co-authored-by: Dubberman <48425266+whipser030@users.noreply.github.com> --------- Co-authored-by: Memtensor-AI <project@memtensor.cn> Co-authored-by: MemOS AutoDev <autodev@memtensor.ai> Co-authored-by: Matthew <heimixiaozhuang@zju.edu.cn> Co-authored-by: MemOS AutoDev <autodev@memos.ai> Co-authored-by: MemOS AutoDev Bot <autodev@memtensor.local> Co-authored-by: Jiang <33757498+hijzy@users.noreply.github.com> Co-authored-by: GU TIANCHUN <96930846+TianchunGu@users.noreply.github.com> Co-authored-by: Dubberman <48425266+whipser030@users.noreply.github.com>
…nsor#2145) This reverts commit 21a27ed. Co-authored-by: jiachengzhen <jiacz@memtensor.cn>
Fixes MemTensor#1391 When causes the install script to exit early (e.g. npm install fails, tarball extraction fails, etc.), the OpenClaw gateway was already stopped but never restarted, leaving the service down. Add an ERR trap after in both install scripts that restarts the gateway on failure. The trap is cleared once the normal step succeeds.
7ee646c to
90c53de
Compare
🤖 Open Code ReviewTarget: PR #2152 🔍 OpenCodeReview found 2 issue(s) in this PR. 1.
|
|
Description
When
set -euo pipefailcauses the install script to exit early (e.g. npm install fails, tarball extraction fails), the OpenClaw gateway was already stopped but never restarted, leaving the service down.This PR adds an ERR trap after
gateway stopin both install scripts (memos-local-plugin/install.shandmemos-local-openclaw/install.sh) that restarts the gateway on failure. The trap is cleared once the normalgateway startstep succeeds.Related Issue: Fixes #1391
Type of change
How Has This Been Tested
bash -nsyntax check passes for both scriptsChecklist