Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 29 additions & 38 deletions packages/agentmask/openclaw-plugin-discovery/VALIDATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ plus its sample-services daemon on the laptop.
│ relay (libp2p) — writes ~/.libp2p-relay/relay.addr │
│ │
│ matcher daemon — OCAP_HOME=~/.ocap (default) │
│ (subcluster running the matcher vat,
plus the llm-bridge process)
│ (subcluster running the matcher vat; the daemon
registers a languageModelService from llm.json)
│ │
│ consumer daemon — OCAP_HOME=~/.ocap-consumer │
│ (no subclusters; hosts the openclaw plugin's │
Expand Down Expand Up @@ -81,10 +81,12 @@ Four things must be live before starting this validation:
[Restarting](#restarting) section below for what this command
actually does and when you should use a different one.

The matcher ranks via an LLM through `@ocap/llm-bridge`, which
talks to the openclaw gateway's OpenAI-compatible
`/v1/chat/completions` endpoint. That adds two requirements,
described in the next two subsections.
The matcher ranks via the daemon's `languageModelService`
kernel service, which talks to the openclaw gateway's
OpenAI-compatible `/v1/chat/completions` endpoint
(`start-matcher.sh` provisions the daemon's `~/.ocap/llm.json`
for this). That adds two requirements, described in the next
two subsections.

3. **Sample-services daemon** on the laptop (the same machine as the
MetaMask extension), started via
Expand Down Expand Up @@ -129,7 +131,7 @@ already exists** — copy the existing value and reuse it; clobbering
the token would invalidate any other clients pointed at the gateway.
Only the `chatCompletions` endpoint flag is reliably new.

### Bridge env vars
### LLM env vars

In the shell that runs `start-matcher.sh` (set these in your shell
profile alongside `LIBP2P_RELAY_PUBLIC_IP`):
Expand All @@ -141,25 +143,15 @@ export OPENCLAW_GATEWAY_TOKEN=<the existing gateway.auth.token value>
# export OPENCLAW_AGENT_MODEL=openclaw # default
```

The bridge runs as a background process beside the matcher daemon
(pid file at `~/.ocap/matcher-llm-bridge.pid`, log at
`~/.ocap/matcher-llm-bridge.log`). `start-matcher.sh` reaps any
previous bridge before spawning a new one; `reset-everything.sh`
tears it down alongside the daemons.

The bridge log captures both halves of every round trip so you can
see exactly what's flowing to and from the LLM. Each ingest or
query produces:

```text
[llm-bridge] -> ingest: {full request JSON}
[llm-bridge] → chat-completions request: {full messages array sent to openclaw}
[llm-bridge] ← chat-completions reply: {full response body from openclaw}
[llm-bridge] <- ingested (svc:0)
```
The LLM calls run inside the matcher daemon itself (no separate
bridge process). The daemon reads `~/.ocap/llm.json` at startup and
registers the `languageModelService` kernel service; the matcher vat
sends one chat-completion request per `findServices` call, carrying
the full current registry in the prompt. Registrations are purely
local to the vat and involve no LLM traffic.

`tail -f ~/.ocap/matcher-llm-bridge.log` while you exercise the
matcher to watch prompts and replies in real time.
`tail -f ~/.ocap/daemon.log` while you exercise the matcher to watch
ranking activity (the matcher vat logs with a `[matcher]` prefix).

## Restarting

Expand Down Expand Up @@ -355,15 +347,15 @@ pre-configured.

Expected tool: `discovery_find_services(description: "sign a
message with my wallet")`. Expected response: PersonalMessageSigner
as the top candidate. The LLM bridge ranks candidates against the
as the top candidate. The LLM ranks candidates against the
query, so the exact response shape depends on the model's output,
but a competently-configured model should pick PMS clearly and
either omit Echo/RandomNumber or rank them well below it. Each
returned candidate carries a `contact (public): ocap:…` URL plus
a `rationale` string in the model's own words. If you see a
"bridge query error" or "bridge ingest error" message instead,
check `~/.ocap/matcher-llm-bridge.log` and the openclaw gateway
config (token + chatCompletions endpoint enabled).
ranking error propagated from the LLM instead, check
`~/.ocap/daemon.log` and the openclaw gateway config (token +
chatCompletions endpoint enabled).

3. **Inspect PMS.**

Expand Down Expand Up @@ -454,15 +446,14 @@ Light-touch. Prompts the agent to hit edge cases.

## Known limitations going in

- Matcher `findServices` uses an LLM-backed Stage-2 ranker via
`@ocap/llm-bridge`, which calls openclaw's
`/v1/chat/completions`. The bridge is started by
`start-matcher.sh` and writes to `~/.ocap/matcher-llm-bridge.log`.
Bridge errors propagate to the consumer rather than falling back
to a heuristic ranker — silent fallbacks would hide LLM-side
problems during development. Stage-3 RAG-style indexing is the
next planned step; today every registration's full digest sits
in the matcher's LLM context window.
- Matcher `findServices` uses an LLM-backed Stage-2 ranker via the
daemon's `languageModelService` kernel service, which calls
openclaw's `/v1/chat/completions`. LLM errors propagate to the
consumer rather than falling back to a heuristic ranker — silent
fallbacks would hide LLM-side problems during development.
Stage-3 RAG-style indexing is the next planned step; today every
registered service's full digest rides along in each ranking
prompt.
- Matcher URL is stable across plain daemon restarts of the same
OCAP home (durable `publicFacet` kref + persisted peer ID and
encryption key), but **re-running `start-matcher.sh` allocates a
Expand Down
2 changes: 1 addition & 1 deletion packages/kernel-agents/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -194,10 +194,10 @@
"dependencies": {
"@endo/eventual-send": "^1.3.4",
"@metamask/kernel-errors": "workspace:^",
"@metamask/kernel-language-model-service": "workspace:^",
"@metamask/kernel-utils": "workspace:^",
"@metamask/logger": "workspace:^",
"@metamask/superstruct": "^3.2.1",
"@ocap/kernel-language-model-service": "workspace:^",
"partial-json": "^0.1.7",
"ses": "^1.14.0"
}
Expand Down
2 changes: 1 addition & 1 deletion packages/kernel-agents/src/agent.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import type { LanguageModel } from '@metamask/kernel-language-model-service';
import { mergeDisjointRecords } from '@metamask/kernel-utils';
import type { Logger } from '@metamask/logger';
import type { LanguageModel } from '@ocap/kernel-language-model-service';

import { doAttempt } from './attempt.ts';
import { TaskManager } from './task.ts';
Expand Down
2 changes: 1 addition & 1 deletion packages/kernel-agents/src/attempt.test.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import '@ocap/repo-tools/test-utils/mock-endoify';

import type { LanguageModel } from '@metamask/kernel-language-model-service';
import type { Logger } from '@metamask/logger';
import type { LanguageModel } from '@ocap/kernel-language-model-service';
import { vi, describe, it, expect, beforeEach } from 'vitest';

import { doAttempt } from './attempt.ts';
Expand Down
2 changes: 1 addition & 1 deletion packages/kernel-agents/src/attempt.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { SampleGenerationError } from '@metamask/kernel-errors';
import type { LanguageModel } from '@metamask/kernel-language-model-service';
import type { Logger } from '@metamask/logger';
import type { LanguageModel } from '@ocap/kernel-language-model-service';

import type { Message, MessageTypeBase } from './types/messages.ts';
import type { PREP, Progress } from './types.ts';
Expand Down
2 changes: 1 addition & 1 deletion packages/kernel-agents/src/strategies/chat-agent.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import type {
ChatMessage,
ChatResult,
ToolCall,
} from '@ocap/kernel-language-model-service';
} from '@metamask/kernel-language-model-service';
import { describe, expect, it, vi } from 'vitest';

import { makeChatAgent } from './chat-agent.ts';
Expand Down
6 changes: 3 additions & 3 deletions packages/kernel-agents/src/strategies/chat-agent.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
import type { Logger } from '@metamask/logger';
import type {
ChatMessage,
ChatResult,
Tool,
} from '@ocap/kernel-language-model-service';
import { parseToolArguments } from '@ocap/kernel-language-model-service/utils/parse-tool-arguments';
} from '@metamask/kernel-language-model-service';
import { parseToolArguments } from '@metamask/kernel-language-model-service/utils/parse-tool-arguments';
import type { Logger } from '@metamask/logger';

import { extractCapabilitySchemas } from '../capabilities/capability.ts';
import { validateCapabilityArgs } from '../capabilities/validate-capability-args.ts';
Expand Down
1 change: 1 addition & 0 deletions packages/kernel-cli/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- The daemon registers a `languageModelService` kernel service at startup when `llm.json` is present in the OCAP home, backed by any Open /v1-compatible endpoint (`provider`, `baseUrl`, and an API key indirected through `apiKeyEnv` or `apiKeyFile`); subclusters request it by listing the service name in their cluster config ([#955](https://github.com/MetaMask/ocap-kernel/pull/955))
- Add global `--home <dir>` flag overriding `$OCAP_HOME` for the duration of one invocation, so multiple OCAP daemons can run side by side without juggling environment variables ([#952](https://github.com/MetaMask/ocap-kernel/pull/952))
- Add `--public-ip <addr>` to `kernel relay start` (also reads `$LIBP2P_RELAY_PUBLIC_IP`); the relay announces the supplied IPv4 alongside its bound NIC addresses, so a NAT-backed VPS can be reached from off-host peers ([#952](https://github.com/MetaMask/ocap-kernel/pull/952))
- More legible output from `kernel relay status` ([#952](https://github.com/MetaMask/ocap-kernel/pull/952))
Expand Down
2 changes: 2 additions & 0 deletions packages/kernel-cli/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -47,10 +47,12 @@
},
"dependencies": {
"@endo/promise-kit": "^1.1.13",
"@metamask/kernel-language-model-service": "workspace:^",
"@metamask/kernel-node-runtime": "workspace:^",
"@metamask/kernel-shims": "workspace:^",
"@metamask/kernel-utils": "workspace:^",
"@metamask/logger": "workspace:^",
"@metamask/superstruct": "^3.2.1",
"@metamask/utils": "^11.9.0",
"@types/node": "^22.13.1",
"acorn": "^8.15.0",
Expand Down
9 changes: 9 additions & 0 deletions packages/kernel-cli/src/commands/daemon-entry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import { join } from 'node:path';

import { getOcapHome } from '../ocap-home.ts';
import { isProcessAlive } from '../utils.ts';
import { makeLlmKernelService, readLlmConfig } from './llm-config.ts';

main().catch((error) => {
process.stderr.write(`Daemon fatal: ${String(error)}\n`);
Expand Down Expand Up @@ -59,6 +60,14 @@ async function main(): Promise<void> {
let handle: DaemonHandle;
try {
await kernel.initIdentity();
const llmConfig = await readLlmConfig(ocapDir);
if (llmConfig) {
const { name, service } = await makeLlmKernelService(llmConfig);
kernel.registerKernelServiceObject(name, service);
logger.info(
`Registered kernel service "${name}" (${llmConfig.provider} at ${llmConfig.baseUrl})`,
);
}
await writeFile(pidPath, String(process.pid));

handle = await startDaemon({
Expand Down
140 changes: 140 additions & 0 deletions packages/kernel-cli/src/commands/llm-config.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
import '@metamask/kernel-shims/endoify';
import { mkdtemp, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';

import {
LLM_CONFIG_FILENAME,
makeLlmKernelService,
readLlmConfig,
resolveLlmApiKey,
} from './llm-config.ts';

describe('llm-config', () => {
let ocapDir: string;

beforeEach(async () => {
ocapDir = await mkdtemp(join(tmpdir(), 'llm-config-test-'));
});

afterEach(async () => {
vi.unstubAllEnvs();
await rm(ocapDir, { recursive: true, force: true });
});

/**
* Write an `llm.json` into the test OCAP home.
*
* @param contents - The raw file contents.
*/
async function writeConfig(contents: string): Promise<void> {
await writeFile(join(ocapDir, LLM_CONFIG_FILENAME), contents);
}

describe('readLlmConfig', () => {
it('returns undefined when no config file exists', async () => {
expect(await readLlmConfig(ocapDir)).toBeUndefined();
});

it('returns the parsed config', async () => {
await writeConfig(
JSON.stringify({
provider: 'open-v1',
baseUrl: 'http://127.0.0.1:18789',
apiKeyEnv: 'TEST_LLM_TOKEN',
}),
);
expect(await readLlmConfig(ocapDir)).toStrictEqual({
provider: 'open-v1',
baseUrl: 'http://127.0.0.1:18789',
apiKeyEnv: 'TEST_LLM_TOKEN',
});
});

it('throws on invalid JSON', async () => {
await writeConfig('{not json');
await expect(readLlmConfig(ocapDir)).rejects.toThrow(/Invalid JSON/u);
});

it.each([
['an unknown provider', { provider: 'closed-v9', baseUrl: 'x' }],
['a missing baseUrl', { provider: 'open-v1' }],
[
'an unknown key',
{ provider: 'open-v1', baseUrl: 'x', apiKey: 'inline-secret' },
],
])('throws on %s', async (_case, config) => {
await writeConfig(JSON.stringify(config));
await expect(readLlmConfig(ocapDir)).rejects.toThrow(
/Invalid LLM config/u,
);
});
});

describe('resolveLlmApiKey', () => {
it('reads the key from the named env var', async () => {
vi.stubEnv('TEST_LLM_TOKEN', 'sekrit');
expect(
await resolveLlmApiKey({
provider: 'open-v1',
baseUrl: 'x',
apiKeyEnv: 'TEST_LLM_TOKEN',
}),
).toBe('sekrit');
});

it('throws when the named env var is unset', async () => {
await expect(
resolveLlmApiKey({
provider: 'open-v1',
baseUrl: 'x',
apiKeyEnv: 'TEST_LLM_TOKEN_UNSET',
}),
).rejects.toThrow(/unset or empty/u);
});

it('reads the key from the named file, trimmed', async () => {
const keyPath = join(ocapDir, 'token.txt');
await writeFile(keyPath, ' file-sekrit\n');
expect(
await resolveLlmApiKey({
provider: 'open-v1',
baseUrl: 'x',
apiKeyFile: keyPath,
}),
).toBe('file-sekrit');
});

it('throws when the named file is empty', async () => {
const keyPath = join(ocapDir, 'token.txt');
await writeFile(keyPath, '\n');
await expect(
resolveLlmApiKey({
provider: 'open-v1',
baseUrl: 'x',
apiKeyFile: keyPath,
}),
).rejects.toThrow(/is empty/u);
});

it('returns undefined when the config names no key source', async () => {
expect(
await resolveLlmApiKey({ provider: 'open-v1', baseUrl: 'x' }),
).toBeUndefined();
});
});

describe('makeLlmKernelService', () => {
it('builds a registrable languageModelService', async () => {
vi.stubEnv('TEST_LLM_TOKEN', 'sekrit');
const { name, service } = await makeLlmKernelService({
provider: 'open-v1',
baseUrl: 'http://127.0.0.1:18789',
apiKeyEnv: 'TEST_LLM_TOKEN',
});
expect(name).toBe('languageModelService');
expect(service).toHaveProperty('chat');
});
});
});
Loading
Loading