Skip to content

fix: ensure gateway restarts on install failure via ERR trap#2152

Open
Zhe-SH-CN wants to merge 21 commits into
MemTensor:mainfrom
Zhe-SH-CN:fix/install-trap-gateway-restart
Open

fix: ensure gateway restarts on install failure via ERR trap#2152
Zhe-SH-CN wants to merge 21 commits into
MemTensor:mainfrom
Zhe-SH-CN:fix/install-trap-gateway-restart

Conversation

@Zhe-SH-CN

Copy link
Copy Markdown

Description

When set -euo pipefail causes the install script to exit early (e.g. npm install fails, tarball extraction fails), the OpenClaw gateway was already stopped but never restarted, leaving the service down.

This PR adds an ERR trap after gateway stop in both install scripts (memos-local-plugin/install.sh and memos-local-openclaw/install.sh) that restarts the gateway on failure. The trap is cleared once the normal gateway start step succeeds.

Related Issue: Fixes #1391

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested

  • bash -n syntax check passes for both scripts
  • Manual code review confirming trap lifecycle (set after stop, cleared after start)

Checklist

  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have linked the issue to this PR

Zyntrael and others added 17 commits July 8, 2026 16:13
* feat(api): align OpenMem v1 SDK with cloud API
Co-authored-by: shinetata <149466187+shinetata@users.noreply.github.com>
…#2113)

Rewrite scheduler Quick Start examples to call the real /product/scheduler REST endpoints with requests instead of non-existent MemOSClient methods; align response fields with the actual handler output; rename the mis-named ' wait.md' to 'wait.md'.

Closes MemTensor#2083

Co-authored-by: sunqi <sunqi@memtensor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ensor#2112)

fix(api): show structured add example in /docs (messages not string)

Swagger UI rendered APIADDRequest.messages as "string" because it picks the
leading `str` branch of the `str | MessageList | RawMessageList` union when
generating the example. Add a model-level json_schema_extra example so /docs
shows a copy-paste-ready payload with a structured messages list.

Schema-only change: field types, validation and runtime behaviour are
unchanged, so the core add pipeline is unaffected.

Closes MemTensor#1505

Co-authored-by: sunqi <sunqi@memtensor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…emTensor#2096)

The hermes adapter's __init__.py imported MemosHttpClient from
bridge_client, but the class was never committed — every plugin python
test failed at collection and any real Hermes host loading memos_provider
hit ImportError at module load. The phantom usages arrived via the
pr/may27-fixes merge (0398e0e) as a half-committed HTTP-bridge feature.

Revert the four usages and the HTTP-first branches in initialize /
_reconnect_bridge, remove the _connect_http_bridge helper and the now-
unused probe_viewer_status / startup_lock_active imports (their
definitions in daemon_manager.py are untouched for a future HTTP PR).
Add test_module_imports_cleanly regression test asserting
MemTensorProvider / MemosBridgeClient / BridgeError are present on the
real contract surfaces and MemosHttpClient is absent on both
memos_provider and bridge_client, so this class of import-level
breakage cannot silently return.

Restores the adapter to stdio-only behavior. Net diff: 57 additions /
54 deletions across the adapter plus the regression test.

Fixes MemTensor#2096
Avoid re-running dictConfig on every get_logger call, since repeated configuration can close and replace active handlers while background threads are emitting records. Keep logging configuration process-local and reconfigure only after a PID change.
Use FastAPI lifespan cleanup to stop scheduler and RabbitMQ resources before logging handlers are torn down during worker or pod shutdown.
## Description

fix: configure logging once per process
fix: close scheduler resources on API shutdown

## Type of change

- [x] Bug fix (non-breaking change which fixes an issue)

## How Has This Been Tested?

- [x] Unit Test - tests/api/test_lifecycle.py, tests/test_log.py

## Checklist

- [x] I have performed a self-review of my own code | 我已自行检查了自己的代码
- [x] I have commented my code in hard-to-understand areas |
我已在难以理解的地方对代码进行了注释
- [x] I have added tests that prove my fix is effective or that my
feature works | 我已添加测试以证明我的修复有效或功能正常
- [ ] I have created related documentation issue/PR in
[MemOS-Docs](https://github.com/MemTensor/MemOS-Docs) (if applicable) |
我已在 [MemOS-Docs](https://github.com/MemTensor/MemOS-Docs) 中创建了相关的文档
issue/PR(如果适用)
- [x] I have linked the issue to this PR (if applicable) | 我已将 issue
链接到此 PR(如果适用)
- [x] I have mentioned the person who will review this PR | 我已提及将审查此 PR
的人

## Reviewer Checklist
- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] Made sure Checks passed
- [ ] Tests have been provided
@Memtensor-AI Memtensor-AI added area:plugin OpenClaw & Hermes status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Jul 23, 2026
@Memtensor-AI
Memtensor-AI requested review from hijzy and whipser030 July 23, 2026 10:36
harvey_xiang and others added 4 commits July 23, 2026 19:07
* Fix MemTensor#1540: fix: (MemTensor#1824)

docs(memos-local-plugin): clarify install path and stale dir names (MemTensor#1540)

The README's 'Quick start' section told users to use install.sh instead
of npm install, but the warning was buried and users still tried
'npm install -g @memtensor/memos-local-plugin' first. The reporter in
MemTensor#1540 encountered this on a Hermes deployment.

This change:

- Promotes the 'do not run npm install -g' notice to a prominent
  IMPORTANT callout explaining why global install is wrong (no
  agent-home deploy, no config.yaml, no bridge/viewer) and that the
  tarball intentionally ships built artifacts only.
- Adds a Troubleshooting subsection covering the two specific symptoms
  in the bug report: the 'package not found' misread, and the stale
  web/ and site/ directory names (web/ is now viewer/, site/ was
  removed by commit 26e7e3d).
- Mentions install.ps1 for Windows alongside install.sh.
- CHANGELOG: record the docs fix and reference MemTensor#1540.

Documentation-only change; no code or runtime behavior touched.

Co-authored-by: MemOS AutoDev <autodev@memtensor.ai>
Co-authored-by: Matthew <heimixiaozhuang@zju.edu.cn>

* Fix MemTensor#1888: [Bug] test_system_parser.py: SystemParser.__init__() got an unexpected keyword a (MemTensor#1889)

fix: remove invalid chunker parameter from SystemParser test instantiation

- SystemParser.__init__() signature changed to (embedder, llm=None)
- Test was still passing chunker=None causing TypeError
- Fixes all 5 failing tests in test_system_parser.py

Fixes MemTensor#1888

Co-authored-by: MemOS AutoDev <autodev@memos.ai>
Co-authored-by: Matthew <heimixiaozhuang@zju.edu.cn>

* Fix MemTensor#1525: [Bug] clean_json_response crashes with cryptic AttributeError when given None (MemTensor#1884)

* test: add comprehensive tests for clean_json_response (issue MemTensor#1525)

- Add test suite in tests/mem_os/test_format_utils.py
- Cover None input ValueError with diagnostic message
- Cover markdown removal, whitespace stripping, edge cases
- Verify fix for AttributeError when LLM returns None

* style: format clean_json_response tests

---------

Co-authored-by: MemOS AutoDev <autodev@memos.ai>
Co-authored-by: Matthew <heimixiaozhuang@zju.edu.cn>

* Fix MemTensor#1901: share_cube_with_user passes swapped args to _validate_cube_access — fails for ev (MemTensor#1903)

fix: validate current user not target in share_cube_with_user (MemTensor#1901)

share_cube_with_user(cube_id, target_user_id) called
_validate_cube_access(cube_id, target_user_id), but the validator
signature is (user_id, cube_id). The cube_id therefore landed in the
user_id slot and _validate_user_exists raised
"User '<cube_id>' does not exist or is inactive" for every well-formed
call, making the API unusable.

The in-code comment "Validate current user has access to this cube"
already documented the correct intent: the sharing user (self.user_id)
must have access to the cube being shared, not the target. Switch the
call to self._validate_cube_access(self.user_id, cube_id). The target
user's existence is independently checked on the next line via
validate_user(target_user_id), so that path is unchanged.

Add regression tests in tests/mem_os/test_memos_core.py that pin down:
- validate_user_cube_access is consulted with (self.user_id, cube_id),
- add_user_to_cube is called with (target_user_id, cube_id) on success,
- a missing target raises "Target user '<id>' does not exist".

Closes MemTensor#1901

Co-authored-by: MemOS AutoDev Bot <autodev@memtensor.local>
Co-authored-by: Matthew <heimixiaozhuang@zju.edu.cn>

* feat(memos): chunk batch reflection scoring

* Fix MemTensor#1897: bug(memos-local-plugin): Hermes background model status checks can trigger paid  (MemTensor#1899)

* Fix MemTensor#1897: fix(memos-local-plugin): add LLM circuit breaker for terminal provider errors

Issue MemTensor#1897 reported ~12,900 paid LLM requests in 24 h on Hermes
against a DeepSeek key with insufficient balance. The local
`system_model_status` row count (12,900) closely tracked the
provider-side `request_count` (11,344) for the same billing window.
The naming is misleading: `system_model_status` is not a health probe;
it is the audit row written once per LLM call (ok / fallback / error)
inside `core/llm/client.ts`. With no circuit breaker, every pipeline
subscriber (capture / session-relation / reward / L2 / L3 / skill /
retrieval LLM filter / world-model) kept firing on every turn / closed
episode / induction, generating one paid request each.

Add a per-`LlmClient` circuit breaker:

- Trips on terminal errors: HTTP 401/402/403 or messages containing
  `insufficient balance` / `invalid api key` / `unauthorized` /
  `account suspended` / `billing`.
- Open: short-circuits subsequent calls inside the facade without
  contacting the provider. Throws `MemosError(LLM_UNAVAILABLE)` with
  `details.circuitOpen=true` so existing catch blocks still work.
- Half-open after cool-down (default 5 min, configurable, min 30 s):
  next call probes the provider; success closes the breaker, terminal
  failure re-opens it for another cool-down.
- Host fallback rescues a call without tripping the breaker —
  fallback exists precisely to keep going when the primary is down.
- Coalesces `system_model_status="circuit_open"` audit rows to at
  most one per ~25 s while the breaker stays open, so we don't
  replace paid spam with audit-row spam.
- Exposes `circuitOpen` / `circuitOpenUntil` / `circuitOpenedReason`
  via `LlmClientStats` for the Overview viewer card.
- Enabled by default; legacy behaviour available via
  `circuitBreaker.enabled = false`.

Tests: 9 new vitest cases under `tests/unit/llm/client.test.ts`
covering trip on 402, trip on "insufficient balance" message, no
trip on generic transient, coalescing, half-open close on success,
host-fallback rescues without trip, disabled mode, stats fields,
and re-open on terminal probe failure. All 59 LLM and 28 pipeline
tests pass; `tsc --noEmit` clean.

Out of scope (tracked separately): 429 `Retry-After` handling
(issue MemTensor#1620), per-tool rate limits, daily budget caps.

* Fix LLM breaker with host fallback

---------

Co-authored-by: autodev-bot <autodev@memtensor.local>
Co-authored-by: Jiang <33757498+hijzy@users.noreply.github.com>
Co-authored-by: GU TIANCHUN <96930846+TianchunGu@users.noreply.github.com>
Co-authored-by: Dubberman <48425266+whipser030@users.noreply.github.com>

---------

Co-authored-by: Memtensor-AI <project@memtensor.cn>
Co-authored-by: MemOS AutoDev <autodev@memtensor.ai>
Co-authored-by: Matthew <heimixiaozhuang@zju.edu.cn>
Co-authored-by: MemOS AutoDev <autodev@memos.ai>
Co-authored-by: MemOS AutoDev Bot <autodev@memtensor.local>
Co-authored-by: Jiang <33757498+hijzy@users.noreply.github.com>
Co-authored-by: GU TIANCHUN <96930846+TianchunGu@users.noreply.github.com>
Co-authored-by: Dubberman <48425266+whipser030@users.noreply.github.com>
…nsor#2145)

This reverts commit 21a27ed.

Co-authored-by: jiachengzhen <jiacz@memtensor.cn>
Fixes MemTensor#1391

When  causes the install script to exit early
(e.g. npm install fails, tarball extraction fails, etc.), the
OpenClaw gateway was already stopped but never restarted, leaving
the service down.

Add an ERR trap after  in both install scripts that
restarts the gateway on failure. The trap is cleared once the
normal  step succeeds.
@Zhe-SH-CN
Zhe-SH-CN force-pushed the fix/install-trap-gateway-restart branch from 7ee646c to 90c53de Compare July 23, 2026 11:07
@Zhe-SH-CN
Zhe-SH-CN changed the base branch from main to dev-v2.0.24 July 23, 2026 11:07
@Memtensor-AI Memtensor-AI added area:api 云服务 / FastAPI / OpenAPI / MCP area:core MOS 编排层 / 框架底座 / 跨模块问题 labels Jul 23, 2026
@Memtensor-AI Memtensor-AI added the area:docs 文档、示例 label Jul 23, 2026
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2152
Task: aaea6be5fdae958c
Base: dev-v2.0.24
Head: fix/install-trap-gateway-restart

🔍 OpenCodeReview found 2 issue(s) in this PR.

⚠️ 1 warning(s) occurred during review.


1. apps/memos-local-openclaw/install.sh (L323)

The ERR trap is set after gateway stop, but line 341 sets a separate EXIT trap (rm -rf "${TMP_PACK_DIR}") later. In bash, setting a new trap on a signal replaces any previously set trap for that signal — but more critically, trap - ERR at the end only clears the ERR trap; the EXIT trap is unaffected, which is correct. However, the ERR trap itself has a subtle issue: if the script is run with set -e (errexit), a failing command triggers ERR and then the shell exits. In that scenario the ERR trap fires gateway start, but the EXIT trap also fires afterward — potentially in an inconsistent order depending on shell behavior. More importantly, if gateway start itself fails inside the ERR trap (non-zero exit), the || true suppresses it, but the ERR trap re-firing is possible in some shells. Consider using a boolean flag pattern instead for more predictable behavior:

_gateway_needs_restart=1
_restart_gateway_on_exit() {
  [[ "$_gateway_needs_restart" == 1 ]] && "${OPENCLAW_BIN}" gateway start >/dev/null 2>&1 || true
}
trap '_restart_gateway_on_exit' EXIT
# ... at the normal success path:
_gateway_needs_restart=0

This avoids ERR-trap re-entrancy issues and integrates cleanly with the existing EXIT trap.


2. apps/memos-local-openclaw/install.sh (L412-L414)

gateway start is not guarded — if it fails (non-zero exit), the ERR trap fires and attempts another gateway start, then the script likely exits. The trap - ERR on the next line never executes in that case, leaving the trap technically irrelevant for that failure path. The start command should be checked explicitly before clearing the trap:

if ! "${OPENCLAW_BIN}" gateway start 2>&1; then
  error "Failed to start OpenClaw Gateway"
  exit 1
fi
trap - ERR

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: INCONCLUSIVE

Automated tests inconclusive (auto-generated test defect); treated as non-blocking. Manual review recommended. Details: Test failures are unrelated to the diff, which only modifies install.sh scripts. The failures are pre-existing test defects: one has an incorrect status assertion, and another references a non-existent 'role' column in the traces table.

Branch: fix/install-trap-gateway-restart

@syzsunshine219
syzsunshine219 changed the base branch from dev-v2.0.24 to main July 23, 2026 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:api 云服务 / FastAPI / OpenAPI / MCP area:core MOS 编排层 / 框架底座 / 跨模块问题 area:docs 文档、示例 area:plugin OpenClaw & Hermes status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix:

8 participants