From 1784e902ade2b49c11c263cc888a147916e0bb10 Mon Sep 17 00:00:00 2001 From: "v.kovalskii" Date: Tue, 17 Mar 2026 17:03:47 +0300 Subject: [PATCH 1/3] Rewrite README: honest positioning, remove bloat, add mcp2cli comparison - Rewrite positioning from "CLI vs MCP" to "CLI = runtime layer" alongside MCP (infrastructure) and Skills (agent knowledge) - Remove implementation details from README: profiles.ini format, caching internals, command mapping rules, architecture section, CLI commands reference (all available via --help) - README reduced from 285 to 172 lines (-40%) - Add mcp2cli to comparison table with MCP/GraphQL/TOON/OAuth features - Add OpenClaw skill with clawhub install command - Replace inline benchmark dump with compact summary + run command - Add fair benchmark note about MCP+Search Compact comparison - Include 4-strategy benchmark (from feat/openclaw-skill-and-comparison) - Include OpenClaw skill files Co-Authored-By: Claude Opus 4.6 (1M context) --- README.md | 324 ++++++------------ benchmarks/benchmark.ts | 720 +++++++++++++++++++++++++++++++++++++++ skills/ocli-api/ISSUE.md | 90 +++++ skills/ocli-api/SKILL.md | 86 +++++ 4 files changed, 1002 insertions(+), 218 deletions(-) create mode 100644 benchmarks/benchmark.ts create mode 100644 skills/ocli-api/ISSUE.md create mode 100644 skills/ocli-api/SKILL.md diff --git a/README.md b/README.md index 6bf5396..03a7bd0 100644 --- a/README.md +++ b/README.md @@ -2,282 +2,170 @@ `openapi-to-cli` (short `ocli`) is a TypeScript CLI that turns any HTTP API described by an OpenAPI/Swagger spec into a set of CLI commands — at runtime, without code generation. -- **Input**: OpenAPI/Swagger spec (URL or file) plus API connection settings. -- **Output**: an executable `ocli` binary where each API operation is exposed as a dedicated subcommand. +```bash +npm install -g openapi-to-cli -Unlike [openapi-to-mcp](https://github.com/EvilFreelancer/openapi-to-mcp), which starts an MCP server with tools, `ocli` provides a direct command-line interface. +ocli profiles add github \ + --api-base-url https://api.github.com \ + --openapi-spec https://api.github.com/openapi.json \ + --api-bearer-token "$GITHUB_TOKEN" -### Why convert OpenAPI spec to CLI? +ocli commands --query "create pull request" --limit 3 +ocli repos_owner_repo_pulls_post --owner octocat --repo hello --title "Fix bug" --head feature --base main +``` -The trend is clear: **CLI commands are cheaper and more native than MCP tools** for AI agents. +### Where CLI fits: MCP, Skills, and CLI -| Factor | MCP Tools | CLI Commands | -|--------|-----------|--------------| -| **Token cost** | Each tool call requires full JSON schema in context on every request | CLI commands are invoked by name with flags — minimal token overhead | -| **Startup overhead** | MCP server must be running, connected via transport layer | Single process, instant execution, zero transport cost | -| **Composability** | Tools are isolated in MCP server scope | CLI commands pipe, chain, and integrate with shell scripts natively | -| **Agent compatibility** | Requires MCP-aware client (Claude, Cursor, etc.) | Any agent that can run shell commands — universal | -| **Discoverability** | Agent must hold all tool schemas in context window | `--help` for quick lookup, `search --query` for BM25-ranked discovery | -| **Multi-API** | One MCP server per API, each consuming context | Multiple profiles in one binary, switch with `ocli use ` | -| **Endpoint scoping** | All tools exposed at once, no per-session filtering | Per-profile `--include/--exclude-endpoints` — same API, different command sets for different roles | -| **Debugging** | Opaque transport, hard to inspect | Plain HTTP requests, visible in terminal | +MCP, skills, and CLI are not competing approaches — they solve different problems at different layers: -When an agent has access to 200+ API endpoints, loading all of them as MCP tools burns thousands of tokens per turn. With `ocli`, the agent calls `ocli search --query "upload files"` to discover relevant commands, then executes them directly. The context window stays clean. +| Layer | Tool | Best for | +|-------|------|----------| +| **Infrastructure** | MCP | Centralized APIs, enterprise SSO, shared state between agents, persistent connections | +| **Agent knowledge** | Skills | On-demand instructions, context isolation, teaching agents _when_ to use a tool | +| **Runtime execution** | CLI | Long action chains, automation, shell pipelines, minimal per-turn token cost | -**Bottom line**: if your agent talks to HTTP APIs, CLI is the most token-efficient and portable interface available today. +`ocli` lives at the **runtime layer**. When an agent needs to call a REST API — search for the right endpoint, check its parameters, execute the call — CLI does this with minimal context overhead and zero infrastructure. -### Comparison +MCP is the right choice when you need centralized auth, persistent connections, or shared state. CLI is the right choice when you need a lightweight, portable way to call HTTP APIs from any agent with shell access. -| Feature | ocli | [openapi-cli-generator](https://github.com/danielgtaylor/openapi-cli-generator) | [CLI-Anything](https://github.com/HKUDS/CLI-Anything) | -|---------|:----:|:---------------------:|:-------------:| -| Runtime interpretation (no codegen) | ✅ | ❌ | ❌ | -| Works without LLM | ✅ | ✅ | ❌ | -| Zero-setup install (`npx`) | ✅ | ❌ | ❌ | -| Instant API onboarding (seconds) | ✅ | ❌ | ❌ | -| Multiple API profiles in one binary | ✅ | ❌ | ❌ | -| Multiple endpoint sets per API | ✅ | ❌ | ❌ | -| BM25 command search | ✅ | ❌ | ❌ | -| Regex command search | ✅ | ❌ | ❌ | -| Per-profile endpoint filtering | ✅ | ❌ | ❌ | -| OpenAPI/Swagger spec (JSON + YAML) | ✅ | ✅ | ❌ | -| Spec caching with refresh | ✅ | ❌ | ❌ | -| Add new API without recompile | ✅ | ❌ | ❌ | -| Non-HTTP integrations (desktop apps) | ❌ | ❌ | ✅ | -| Session management / undo-redo | ❌ | ❌ | ✅ | -| JSON structured output | ❌ | ✅ | ✅ | -| Basic / Bearer auth | ✅ | ✅ | ❌ | -| OAuth2 / Auth0 | ❌ | ✅ | ✅ | -| Response JMESPath filtering | ❌ | ✅ | ❌ | -| Syntax-highlighted output | ❌ | ✅ | ❌ | -| Auto-generated tests | ❌ | ❌ | ✅ | -| Active project | ✅ | ❌ (deprecated) | ✅ | - -### High level idea - -- The user installs the package (for example via `npx` or globally) and gets the `ocli` binary in `$PATH`. -- On first use the user onboards an API with a command like: +### Quick start ```bash -ocli profiles add myapi \ - --api-base-url http://127.0.0.1:2222 \ - --openapi-spec http://127.0.0.1:2222/openapi.json \ - --api-bearer-token "..." \ - --include-endpoints get:/messages,get:/channels \ - --exclude-endpoints post:/admin/secret -``` +# Install +npm install -g openapi-to-cli -Alternatively, `ocli onboard` (with the same options, no profile name) creates a profile named `default`. +# Add an API profile +ocli profiles add myapi \ + --api-base-url https://api.example.com \ + --openapi-spec https://api.example.com/openapi.json \ + --api-bearer-token "$TOKEN" \ + --include-endpoints "get:/messages,post:/messages" \ + --command-prefix "myapi_" \ + --custom-headers '{"X-Tenant":"acme"}' + +# Set as active profile +ocli use myapi -- The CLI: - - downloads and validates the OpenAPI spec; - - stores profile configuration; - - caches the spec in the filesystem; - - builds a set of commands based on paths and methods. +# Discover commands +ocli commands --query "send message" --limit 5 -After onboarding, commands can be used like: +# Check parameters +ocli myapi_messages_post --help -```bash -ocli messages_get --profile myapi --limit 10 -ocli channels_username_get --profile myapi --username alice +# Execute +ocli myapi_messages_post --text "Hello world" ``` -or using the default profile: +Or use `npx` without global install: ```bash -ocli use myapi -ocli messages --limit 10 +npx openapi-to-cli onboard \ + --api-base-url https://api.example.com \ + --openapi-spec https://api.example.com/openapi.json ``` ### Command search -When the API surface is too large for `--help`, use command filtering with `commands`: - ```bash # BM25 natural language search -ocli commands --query "upload files" -ocli commands -q "list messages" --limit 5 +ocli commands --query "upload files" --limit 5 # Regex pattern matching -ocli commands --regex "admin.*get" -ocli commands -r "messages" -n 3 -``` - -The BM25 engine (ported from [picoclaw](https://github.com/sipeed/picoclaw)) ranks commands by relevance across name, method, path, description, and parameter names. This enables agents to discover the right endpoint without loading all command schemas into context. The legacy `ocli search` command is kept as a deprecated alias and internally forwards to `ocli commands` with the same flags. +ocli commands --regex "users.*post" --limit 10 -### Installation and usage via npm and npx - -To use `ocli` locally without installing it globally you can rely on `npx`: - -```bash -npx openapi-to-cli onboard \ - --api-base-url http://127.0.0.1:2222 \ - --openapi-spec http://127.0.0.1:2222/openapi.json +# List all commands +ocli commands ``` -The command above will - -- download the `openapi-to-cli` package from npm if it is not cached yet -- run the `ocli` binary from the package -- create the `default` profile and cache the OpenAPI spec under `.ocli/specs/default.json` +The BM25 engine ranks commands by relevance across name, method, path, description, and parameter names. Tested on APIs with 845+ endpoints (GitHub API). -After onboarding you can continue to use the generated commands with the `ocli` binary that `npx` runs for you: +### Using with AI agents -```bash -npx openapi-to-cli use myapi -npx openapi-to-cli messages_get --limit 10 -``` - -If you prefer a global installation you can also install the package once - -```bash -npm install -g openapi-to-cli -``` +#### OpenClaw skill -and then call the binary directly +Install the [ocli-api](https://clawhub.ai/skills/ocli-api) skill from [ClawHub](https://clawhub.ai): ```bash -ocli onboard --api-base-url http://127.0.0.1:2222 --openapi-spec http://127.0.0.1:2222/openapi.json -ocli messages_get --limit 10 +clawhub install ocli-api ``` -### Profiles and configuration files - -- A profile describes a single API connection. -- Profiles are stored in an INI file (one section per profile, no special "current" key in the INI): - - global: `~/.ocli/profiles.ini` - - project-local: `./.ocli/profiles.ini` (has higher priority than global) -- The profile to use when the user does not pass `--profile` is stored in `.ocli/current` (one line: profile name). If the file is missing or empty, the profile named `default` is used. The profile named `default` is a normal profile like any other; it is just used when no profile is specified. - -Example `profiles.ini` structure: - -```ini -[default] -api_base_url = http://127.0.0.1:1111 -api_basic_auth = -api_bearer_token = MY_TOKEN -openapi_spec_source = http://127.0.0.1:1111/openapi.json -openapi_spec_cache = /home/user/.ocli/specs/default.json -include_endpoints = get:/messages,get:/channels -exclude_endpoints = - -[myapi] -api_base_url = http://127.0.0.1:2222 -api_basic_auth = -api_bearer_token = MY_TOKEN -openapi_spec_source = http://127.0.0.1:2222/openapi.json -openapi_spec_cache = /home/user/.ocli/specs/myapi.json -include_endpoints = get:/messages,get:/channels -exclude_endpoints = -``` - -The local file `./.ocli/profiles.ini`, if present, fully overrides the global one when resolving profiles. - -### OpenAPI/Swagger caching +Or manually copy [`skills/ocli-api/SKILL.md`](skills/ocli-api/SKILL.md) to `~/.openclaw/skills/ocli-api/SKILL.md`. -- Config and cache directory: - - globally: `~/.ocli/` - - locally: `./.ocli/` relative to the directory where `ocli` is executed. +#### Claude Code skill -- Inside `.ocli` the CLI creates: - - `profiles.ini` - profile configuration (one section per profile); - - `current` - one line with the profile name to use when `--profile` is not passed (optional; if missing, profile `default` is used); - - `specs/` - directory with cached specs: - - `specs/.json` - OpenAPI spec content for the profile. - -- During onboarding: - - the CLI loads the spec from `--openapi-spec`; - - writes it to `specs/.json`; - - stores the cache path in `openapi_spec_cache` in the corresponding profile section. - -- When running commands: - - by default the spec is read from `openapi_spec_cache`; - - later we can add a flag to force spec refresh (for example `--refresh-spec`) that will overwrite the cache. - -### Mapping OpenAPI operations to CLI commands - -- For each OpenAPI operation (method + path) the CLI exposes one subcommand. -- Command name is derived from the path: - - `/messages` → `messages` - - `/channels/{username}` → `channels_username` -- If the same path segment is used by multiple methods (GET, POST, etc.), a method suffix is added: - - `/messages` GET → `messages_get` - - `/messages` POST → `messages_post` - -Invocation format: +Copy the example skill to your project: ```bash -ocli [--profile ] [options...] +cp examples/skill-ocli-api.md .claude/skills/api.md ``` -where: +#### Agent workflow -- `tool-name` is the name derived from the path (with method suffix when needed); -- `options` is the set of flags representing operation parameters: - - query and path parameters → `--param-name`; - - JSON body fields → also `--field-name`. +1. `ocli commands --query "upload file"` — discover the right command +2. `ocli files_content_post --help` — check parameters +3. `ocli files_content_post --file ./data.csv` — execute -Option types and required flags are determined from the OpenAPI schema. +### Benchmark -### CLI commands and help +Four strategies compared on [Swagger Petstore](https://petstore3.swagger.io/) (19 endpoints), with scaling projections to [GitHub API](https://api.apis.guru/v2/specs/github.com/api.github.com/1.1.4/openapi.json) (845 endpoints). All search strategies use the same BM25 engine. -The `ocli` binary provides the following core commands: +``` + TOOL DEFINITION OVERHEAD (sent with every API request) -- `ocli onboard` - add a new profile named `default` (alias for `ocli profiles add default`). Options: - - `--api-base-url ` - API base URL; - - `--openapi-spec ` - OpenAPI source (URL or file path); - - `--api-basic-auth ` - optional; - - `--api-bearer-token ` - optional; - - `--include-endpoints ` - comma-separated `method:path`; - - `--exclude-endpoints ` - comma-separated `method:path`. + MCP Naive █████████████████████████ 2,945 tok (19 tools) + MCP+Search Full ███ 355 tok (2 tools) + MCP+Search Compact ████ 437 tok (3 tools) + CLI (ocli) █ 158 tok (1 tool) -- `ocli profiles add ` - add a new profile with the given name and cache the OpenAPI spec. Same options as `onboard` (profile name is the positional argument). + TOTAL TOKENS PER TASK (realistic multi-turn agent flow) -- `ocli profiles list` - list all profiles; -- `ocli profiles show ` - show profile details; -- `ocli profiles remove ` - remove a profile; -- `ocli use ` - set the profile to use when `--profile` is not passed (writes profile name to `.ocli/current`). -- `ocli commands` - list available commands generated from the current profile and its OpenAPI spec, optionally filter them with `--query` (BM25) or `--regex`. -- `ocli search` - deprecated alias for `ocli commands` with `--query/--regex`, kept for backward compatibility. -- `ocli --version` - print the CLI version baked at build time (derived from the latest git tag when available). + MCP Naive █████████████████████████ 3,015 tok (1 turn) + MCP+Search Full ██████████████████ 2,185 tok (2 turns) + MCP+Search Compact █████████████████ 2,066 tok (3 turns) + CLI (ocli) ████████ 925 tok (3 turns) -Help: + SCALING: OVERHEAD PER TURN vs ENDPOINT COUNT -- `ocli -h|--help` - global help and command overview; -- `ocli onboard -h|--help` - onboarding help; -- `ocli profiles -h|--help` - profile management help; -- `ocli -h|--help` - description of a particular operation, list of options and their types (generated from OpenAPI). + Endpoints MCP Naive MCP+S Compact CLI (ocli) + 19 2,945 tok 437 tok 158 tok ← Petstore + 845 130,106 tok 437 tok 158 tok ← GitHub API +``` -### Architecture +Run the benchmark yourself: `npx ts-node benchmarks/benchmark.ts` -``` -src/ -├── cli.ts # Entry point, command routing, HTTP requests -├── config.ts # .ocli directory resolution (local > global) -├── openapi-loader.ts # OpenAPI spec download and caching -├── openapi-to-commands.ts # OpenAPI → CLI command generation -├── command-search.ts # BM25 + regex search over commands -├── bm25.ts # BM25 ranking engine (ported from picoclaw) -├── profile-store.ts # Profile persistence in INI format -└── version.ts # Version constant (generated at build) -``` +Note: MCP+Search Compact (search → get_schema → call) is the fairest comparison to CLI (search → --help → execute) — same number of turns, same BM25 engine. The difference is tool definition overhead (437 vs 158 tok/turn) and schema format (JSON vs text). -The project mirrors parts of the `openapi-to-mcp` architecture but implements a CLI instead of an MCP server: +### Comparison -- `config` - reads profile configuration and cache paths (INI files, global and local `.ocli` lookup). -- `profile-store` - works with `profiles.ini` (read, write, select profile, current profile). -- `openapi-loader` - loads and caches the OpenAPI spec (URL or file) into `.ocli/specs/`. -- `openapi-to-commands` - parses OpenAPI, applies include/exclude filters, generates command names and option schemas. -- `command-search` - BM25 and regex search over generated commands for discovery on large API surfaces. -- `bm25` - generic BM25 ranking engine with Robertson IDF smoothing and min-heap top-K extraction. -- `cli` - entry point, argument parser, command registration, help output. +| Feature | ocli | [mcp2cli](https://github.com/knowsuchagency/mcp2cli) | [openapi-cli-generator](https://github.com/danielgtaylor/openapi-cli-generator) | [CLI-Anything](https://github.com/HKUDS/CLI-Anything) | +|---------|:----:|:------:|:---------------------:|:-------------:| +| Runtime interpretation (no codegen) | ✅ | ✅ | ❌ | ❌ | +| Works without LLM | ✅ | ✅ | ✅ | ❌ | +| Zero-setup install (`npx`/`uvx`) | ✅ | ✅ | ❌ | ❌ | +| Multiple API profiles | ✅ | ✅ (bake mode) | ❌ | ❌ | +| BM25 command search | ✅ | ❌ (substring only) | ❌ | ❌ | +| Regex command search | ✅ | ❌ | ❌ | ❌ | +| Per-profile endpoint filtering | ✅ | ✅ | ❌ | ❌ | +| OpenAPI/Swagger (JSON + YAML) | ✅ | ✅ | ✅ | ❌ | +| MCP server support | ❌ | ✅ (HTTP/SSE/stdio) | ❌ | ❌ | +| GraphQL support | ❌ | ✅ (introspection) | ❌ | ❌ | +| Spec caching | ✅ | ✅ (1h TTL) | ❌ | ❌ | +| Custom HTTP headers | ✅ | ✅ | ❌ | ❌ | +| Command name prefix | ✅ | ❌ | ❌ | ❌ | +| Basic / Bearer auth | ✅ | ✅ | ✅ | ❌ | +| OAuth2 | ❌ | ✅ (PKCE) | ✅ | ✅ | +| Response filtering (jq/JMESPath) | ❌ | ✅ (jq) | ✅ (JMESPath) | ❌ | +| Token-optimized output (TOON) | ❌ | ✅ | ❌ | ❌ | +| JSON structured output | ❌ | ✅ | ✅ | ✅ | +| Active project | ✅ | ✅ | ❌ (deprecated) | ✅ | ### Similar projects -- [openapi-cli-generator](https://github.com/danielgtaylor/openapi-cli-generator) - generates a CLI from an OpenAPI 3 specification using code generation. -- [anything-llm-cli](https://github.com/Mintplex-Labs/anything-llm/tree/master/clients/anything-cli) - CLI for interacting with AnythingLLM, can consume HTTP APIs and tools. -- [openapi-commander](https://github.com/bcoughlan/openapi-commander) - Node.js command-line tool generator based on OpenAPI definitions. -- [OpenAPI Generator](https://openapi-generator.tech/docs/usage) - general-purpose OpenAPI code generator that can also generate CLI clients. -- [openapi2cli](https://pypi.org/project/openapi2cli/) - Python tool that builds CLI interfaces for OpenAPI 3 APIs. +- [mcp2cli](https://github.com/knowsuchagency/mcp2cli) — Python CLI that converts MCP servers, OpenAPI specs, and GraphQL endpoints into CLI commands at runtime. Supports OAuth, TOON output format, and daemon sessions. +- [openapi-cli-generator](https://github.com/danielgtaylor/openapi-cli-generator) — generates a CLI from an OpenAPI 3 specification using code generation. +- [openapi-commander](https://github.com/bcoughlan/openapi-commander) — Node.js command-line tool generator based on OpenAPI definitions. +- [OpenAPI Generator](https://openapi-generator.tech/docs/usage) — general-purpose OpenAPI code generator that can also generate CLI clients. +- [openapi2cli](https://pypi.org/project/openapi2cli/) — Python tool that builds CLI interfaces for OpenAPI 3 APIs. ### License diff --git a/benchmarks/benchmark.ts b/benchmarks/benchmark.ts new file mode 100644 index 0000000..911ed96 --- /dev/null +++ b/benchmarks/benchmark.ts @@ -0,0 +1,720 @@ +#!/usr/bin/env ts-node + +/** + * Token benchmark: 4 strategies for AI agent ↔ API interaction + * + * 1. MCP Naive — all endpoints as tools in context (standard MCP approach) + * 2. MCP+Search Full — 2 tools: search_tools (returns full schemas) + call_api + * 3. MCP+Search Compact — 3 tools: search_tools (compact) + get_tool_schema + call_api + * 4. CLI (ocli) — 1 tool: execute_command (search + help + execute) + * + * All search strategies use the same BM25 engine for fair comparison. + * Tested against Swagger Petstore (19 endpoints). + * Run: npx ts-node benchmarks/benchmark.ts + */ + +import axios from "axios"; +import { OpenapiToCommands, CliCommand } from "../src/openapi-to-commands"; +import { CommandSearch } from "../src/command-search"; +import { Profile } from "../src/profile-store"; + +// ── Helpers ────────────────────────────────────────────────────────── + +function estimateTokens(text: string): number { + return Math.ceil(text.length / 4); +} + +function padRight(str: string, len: number): string { + return str.length >= len ? str : str + " ".repeat(len - str.length); +} + +function padLeft(str: string, len: number): string { + return str.length >= len ? str : " ".repeat(len - str.length) + str; +} + +function bar(value: number, max: number, width: number, char = "█"): string { + const filled = Math.round((value / max) * width); + return char.repeat(Math.max(1, filled)); +} + +function matches(value: string, expected: string | RegExp): boolean { + if (typeof expected === "string") return value === expected; + return expected.test(value); +} + +// ── OpenAPI → MCP tool definitions ────────────────────────────────── + +interface McpTool { + name: string; + description: string; + input_schema: Record; +} + +function openapiToMcpTools(spec: Record): McpTool[] { + const tools: McpTool[] = []; + const paths = (spec.paths ?? {}) as Record>; + const schemas = ((spec.components as Record)?.schemas ?? {}) as Record; + const methods = ["get", "post", "put", "delete", "patch"]; + + for (const [pathKey, pathItem] of Object.entries(paths)) { + for (const method of methods) { + const op = pathItem[method] as Record | undefined; + if (!op) continue; + + const operationId = (op.operationId as string) ?? `${method}_${pathKey.replace(/[{}\/]/g, "_")}`; + const description = (op.description as string) ?? (op.summary as string) ?? ""; + const properties: Record = {}; + const required: string[] = []; + + const params = (op.parameters ?? []) as Array>; + for (const param of params) { + const name = param.name as string; + const schema = (param.schema ?? { type: "string" }) as Record; + properties[name] = { + type: schema.type ?? "string", + description: (param.description as string) ?? "", + ...(schema.enum ? { enum: schema.enum } : {}), + ...(schema.format ? { format: schema.format } : {}), + }; + if (param.required) required.push(name); + } + + const requestBody = op.requestBody as Record | undefined; + if (requestBody) { + const content = requestBody.content as Record | undefined; + const jsonContent = content?.["application/json"] as Record | undefined; + const bodySchema = jsonContent?.schema as Record | undefined; + if (bodySchema) { + const ref = bodySchema.$ref as string | undefined; + if (ref) { + const schemaName = ref.split("/").pop()!; + const resolved = schemas[schemaName] as Record | undefined; + if (resolved) { + const bodyProps = (resolved.properties ?? {}) as Record; + const bodyReq = (resolved.required ?? []) as string[]; + for (const [pn, ps] of Object.entries(bodyProps)) { + const prop = ps as Record; + if (prop.$ref) { + const innerName = (prop.$ref as string).split("/").pop()!; + properties[pn] = schemas[innerName] ?? prop; + } else if (prop.items && (prop.items as Record).$ref) { + const innerName = ((prop.items as Record).$ref as string).split("/").pop()!; + properties[pn] = { ...prop, items: schemas[innerName] ?? prop.items }; + } else { + properties[pn] = prop; + } + } + required.push(...bodyReq); + } + } + } + } + + tools.push({ + name: operationId, + description: description.slice(0, 1024), + input_schema: { + type: "object", + properties, + ...(required.length > 0 ? { required } : {}), + }, + }); + } + } + + return tools; +} + +// ── BM25 search over MCP tools (same engine as CLI) ───────────────── + +interface McpSearchable { + tool: McpTool; + tokens: string; +} + +function buildMcpSearchIndex(mcpTools: McpTool[]): McpSearchable[] { + return mcpTools.map(t => { + const propNames = Object.keys((t.input_schema.properties ?? {}) as Record); + return { + tool: t, + tokens: [t.name, t.description, ...propNames].join(" ").toLowerCase(), + }; + }); +} + +/** + * BM25-ranked search over MCP tools. + * Uses the same CommandSearch engine as CLI for fair comparison. + */ +function searchMcpTools( + searcher: CommandSearch, + mcpToolsByName: Map, + query: string, + limit: number, +): McpTool[] { + const results = searcher.search(query, limit); + const matched: McpTool[] = []; + for (const r of results) { + // Match by operationId or by generated command name + const tool = mcpToolsByName.get(r.name) ?? mcpToolsByName.get(r.path); + if (tool) matched.push(tool); + } + // If BM25 didn't find exact matches, fall back to description search + if (matched.length === 0) { + const q = query.toLowerCase(); + const scored = Array.from(mcpToolsByName.values()) + .filter(t => t.name.toLowerCase().includes(q) || t.description.toLowerCase().includes(q)) + .slice(0, limit); + return scored; + } + return matched; +} + +// ── Tool definitions for each strategy ────────────────────────────── + +function buildMcpSearchFullTools(): McpTool[] { + return [ + { + name: "search_tools", + description: "Search available API tools by natural language query. Returns matching tools with their full parameter schemas so you can call them immediately.", + input_schema: { + type: "object", + properties: { + query: { type: "string", description: "Natural language search query (e.g. 'create a pet', 'get order status')" }, + limit: { type: "number", description: "Maximum number of results to return (default: 5)" }, + }, + required: ["query"], + }, + }, + { + name: "call_api", + description: "Call an API endpoint by its tool name with the specified parameters. Use search_tools first to discover available tools and their schemas.", + input_schema: { + type: "object", + properties: { + tool_name: { type: "string", description: "The tool name returned by search_tools (e.g. 'addPet', 'getOrderById')" }, + parameters: { type: "object", description: "Parameters to pass to the tool, matching the schema from search_tools results" }, + }, + required: ["tool_name", "parameters"], + }, + }, + ]; +} + +function buildMcpSearchCompactTools(): McpTool[] { + return [ + { + name: "search_tools", + description: "Search available API tools by natural language query. Returns tool names and descriptions (no parameter schemas).", + input_schema: { + type: "object", + properties: { + query: { type: "string", description: "Natural language search query (e.g. 'create a pet', 'get order status')" }, + limit: { type: "number", description: "Maximum number of results to return (default: 5)" }, + }, + required: ["query"], + }, + }, + { + name: "get_tool_schema", + description: "Get the full parameter schema for a specific tool. Use after search_tools to get the parameters before calling.", + input_schema: { + type: "object", + properties: { + tool_name: { type: "string", description: "The tool name from search_tools results" }, + }, + required: ["tool_name"], + }, + }, + { + name: "call_api", + description: "Call an API endpoint by its tool name with parameters. Use get_tool_schema first to discover required parameters.", + input_schema: { + type: "object", + properties: { + tool_name: { type: "string", description: "The tool name (e.g. 'addPet', 'getOrderById')" }, + parameters: { type: "object", description: "Parameters matching the schema from get_tool_schema" }, + }, + required: ["tool_name", "parameters"], + }, + }, + ]; +} + +function buildCliTool(): McpTool[] { + return [{ + name: "execute_command", + description: "Execute a shell command. Use `ocli commands --query \"...\"` to search for API commands, then `ocli --help` for parameters, then `ocli --param value` to execute.", + input_schema: { + type: "object", + properties: { + command: { type: "string", description: "Shell command to execute" }, + }, + required: ["command"], + }, + }]; +} + +// ── Simulate search results (all using BM25) ──────────────────────── + +/** + * MCP search_tools (full) — returns matched tools with full JSON schemas. + * Uses BM25 ranking, same engine as CLI. + */ +function simulateMcpSearchFullResult( + searcher: CommandSearch, + mcpToolsByName: Map, + query: string, + limit: number, +): string { + const matched = searchMcpTools(searcher, mcpToolsByName, query, limit); + return JSON.stringify(matched.map(t => ({ + name: t.name, + description: t.description, + parameters: t.input_schema, + })), null, 2); +} + +/** + * MCP search_tools (compact) — returns only names and descriptions. + * Same BM25 ranking. Agent must call get_tool_schema separately. + */ +function simulateMcpSearchCompactResult( + searcher: CommandSearch, + mcpToolsByName: Map, + query: string, + limit: number, +): string { + const matched = searchMcpTools(searcher, mcpToolsByName, query, limit); + return JSON.stringify(matched.map(t => ({ + name: t.name, + description: t.description, + })), null, 2); +} + +/** + * MCP get_tool_schema — returns full schema for one tool. + */ +function simulateMcpGetSchemaResult(tool: McpTool): string { + return JSON.stringify({ + name: tool.name, + description: tool.description, + parameters: tool.input_schema, + }, null, 2); +} + +/** + * CLI search — compact text: name + method + path + description. + * Uses the same BM25 engine. + */ +function simulateCliSearchResult(searcher: CommandSearch, query: string, limit: number): string { + const results = searcher.search(query, limit); + return results.map(r => + ` ${r.name.padEnd(35)} ${r.method.padEnd(7)} ${r.path} ${r.description ?? ""}` + ).join("\n"); +} + +/** + * CLI --help — simulates `ocli --help` output. + * Returns command description and parameter list (similar to get_tool_schema). + * Looks up the full CliCommand to get options. + */ +function simulateCliHelpResult(commands: CliCommand[], searcher: CommandSearch, query: string): string { + const results = searcher.search(query, 1); + if (results.length === 0) return "(no command found)"; + const matched = results[0]; + const cmd = commands.find(c => c.name === matched.name); + if (!cmd) return "(no command found)"; + const lines = [ + `${cmd.name} — ${cmd.description ?? ""}`, + ` ${cmd.method.toUpperCase()} ${cmd.path}`, + "", + "Options:", + ]; + for (const opt of cmd.options) { + const req = opt.required ? " (required)" : ""; + lines.push(` --${opt.name.padEnd(20)} ${(opt.schemaType ?? "string").padEnd(10)} ${opt.description ?? ""}${req}`); + } + return lines.join("\n"); +} + +// ── Accuracy tasks ────────────────────────────────────────────────── + +interface AccuracyTask { + query: string; + expected: string | RegExp; + expectedPath: string | RegExp; +} + +const ACCURACY_TASKS: AccuracyTask[] = [ + { query: "find all pets with status available", expected: "pet_findByStatus", expectedPath: "/pet/findByStatus" }, + { query: "add a new pet to the store", expected: /pet.*post/, expectedPath: "/pet" }, + { query: "get pet by id", expected: /pet.*petId.*get/, expectedPath: /\/pet\/\{petId\}/ }, + { query: "update existing pet information", expected: /pet.*put/, expectedPath: "/pet" }, + { query: "delete a pet", expected: /pet.*delete/, expectedPath: /\/pet\/\{petId\}/ }, + { query: "place an order for a pet", expected: /store.*order/, expectedPath: "/store/order" }, + { query: "get store inventory", expected: /store.*inventory/, expectedPath: "/store/inventory" }, + { query: "create a new user account", expected: /^user$/, expectedPath: "/user" }, + { query: "get user by username", expected: /user.*username.*get/, expectedPath: /\/user\/\{username\}/ }, + { query: "find pets by tags", expected: /pet.*findByTags/, expectedPath: "/pet/findByTags" }, + { query: "upload pet photo", expected: /upload/, expectedPath: /uploadImage/ }, + { query: "login to the system", expected: /user.*login/, expectedPath: "/user/login" }, + { query: "check order status", expected: /store.*order.*get/, expectedPath: /\/store\/order\/\{orderId\}/ }, + { query: "remove user from system", expected: /user.*delete/, expectedPath: /\/user\/\{username\}/ }, + { query: "bulk create users", expected: /user.*createWithList/, expectedPath: "/user/createWithList" }, +]; + +// ── Main ──────────────────────────────────────────────────────────── + +async function main(): Promise { + console.log("Fetching Petstore OpenAPI spec...\n"); + const response = await axios.get("https://petstore3.swagger.io/api/v3/openapi.json"); + const spec = response.data as Record; + + // Build MCP tools + const mcpTools = openapiToMcpTools(spec); + const mcpToolsByName = new Map(); + for (const t of mcpTools) mcpToolsByName.set(t.name, t); + + // Build CLI commands + shared BM25 searcher + const profile: Profile = { + name: "petstore", apiBaseUrl: "https://petstore3.swagger.io/api/v3", + apiBasicAuth: "", apiBearerToken: "", openapiSpecSource: "", + openapiSpecCache: "", includeEndpoints: [], excludeEndpoints: [], + commandPrefix: "", customHeaders: {}, + }; + + const commands = new OpenapiToCommands().buildCommands(spec, profile); + const searcher = new CommandSearch(); + searcher.load(commands); + + // Also map CLI command names → MCP tools for cross-referencing + for (const cmd of commands) { + // Try to find MCP tool by path matching + for (const t of mcpTools) { + if (!mcpToolsByName.has(cmd.name)) { + mcpToolsByName.set(cmd.name, t); + } + } + } + + // ── Token calculations for tool definitions (sent every turn) ── + + const mcpNaiveToolsJson = JSON.stringify(mcpTools, null, 2); + const mcpSearchFullToolsJson = JSON.stringify(buildMcpSearchFullTools(), null, 2); + const mcpSearchCompactToolsJson = JSON.stringify(buildMcpSearchCompactTools(), null, 2); + const cliToolJson = JSON.stringify(buildCliTool(), null, 2); + + const mcpNaiveToolTokens = estimateTokens(mcpNaiveToolsJson); + const mcpSearchFullToolTokens = estimateTokens(mcpSearchFullToolsJson); + const mcpSearchCompactToolTokens = estimateTokens(mcpSearchCompactToolsJson); + const cliToolTokens = estimateTokens(cliToolJson); + + // System prompts + const mcpNaiveSystem = "You are an AI assistant with access to the Petstore API. Use the provided tools."; + const mcpSearchFullSystem = "You are an AI assistant. Use search_tools to find API endpoints (returns full schemas), then call_api to execute them."; + const mcpSearchCompactSystem = "You are an AI assistant. Use search_tools to find endpoints, get_tool_schema to get parameters, then call_api to execute."; + const cliSystem = "You are an AI assistant. Use `ocli commands --query` to search, `ocli --help` for parameters, then `ocli --param value` to execute."; + + const mcpNaiveSysTok = estimateTokens(mcpNaiveSystem); + const mcpSearchFullSysTok = estimateTokens(mcpSearchFullSystem); + const mcpSearchCompactSysTok = estimateTokens(mcpSearchCompactSystem); + const cliSysTok = estimateTokens(cliSystem); + + const mcpNaiveOverhead = mcpNaiveToolTokens + mcpNaiveSysTok; + const mcpSearchFullOverhead = mcpSearchFullToolTokens + mcpSearchFullSysTok; + const mcpSearchCompactOverhead = mcpSearchCompactToolTokens + mcpSearchCompactSysTok; + const cliOverhead = cliToolTokens + cliSysTok; + + // ── Measure ACTUAL search results using BM25 for all strategies ── + + const sampleQuery = "find pets by status"; + + const mcpSearchFullResultSample = simulateMcpSearchFullResult(searcher, mcpToolsByName, sampleQuery, 3); + const mcpSearchCompactResultSample = simulateMcpSearchCompactResult(searcher, mcpToolsByName, sampleQuery, 3); + const cliSearchResultSample = simulateCliSearchResult(searcher, sampleQuery, 3); + + // Get schema for one tool (used by compact MCP and CLI --help) + const firstMatchedTool = searchMcpTools(searcher, mcpToolsByName, sampleQuery, 1)[0]; + const mcpGetSchemaResultSample = firstMatchedTool ? simulateMcpGetSchemaResult(firstMatchedTool) : "{}"; + const cliHelpResultSample = simulateCliHelpResult(commands, searcher, sampleQuery); + + const mcpSearchFullResultTok = estimateTokens(mcpSearchFullResultSample); + const mcpSearchCompactResultTok = estimateTokens(mcpSearchCompactResultSample); + const mcpGetSchemaResultTok = estimateTokens(mcpGetSchemaResultSample); + const cliSearchResultTok = estimateTokens(cliSearchResultSample); + const cliHelpResultTok = estimateTokens(cliHelpResultSample); + + console.log(`Petstore API: ${mcpTools.length} endpoints\n`); + + // ══════════════════════════════════════════════════════════════ + // OUTPUT + // ══════════════════════════════════════════════════════════════ + + const W = 80; + const line = "─".repeat(W); + const dline = "═".repeat(W); + + console.log(dline); + console.log(" FOUR STRATEGIES FOR AI AGENT ↔ API INTERACTION"); + console.log(dline); + console.log(); + console.log(" 1. MCP Naive All endpoints as tools in context (1 turn)"); + console.log(" 2. MCP+Search Full search_tools (full schemas) + call_api (2 turns)"); + console.log(" 3. MCP+Search Compact search_tools (compact) + get_schema + call_api (3 turns)"); + console.log(" 4. CLI (ocli) search + --help + execute (3 turns)"); + console.log(); + console.log(" All search strategies use the same BM25 engine for fair comparison."); + console.log(); + + // ── Tool definition overhead ── + console.log(dline); + console.log(" TOOL DEFINITION OVERHEAD (sent with every API request)"); + console.log(dline); + console.log(); + + const maxOvh = mcpNaiveOverhead; + console.log(` MCP Naive ${bar(mcpNaiveOverhead, maxOvh, 25)} ${padLeft(mcpNaiveOverhead.toLocaleString(), 6)} tok (${mcpTools.length} tools)`); + console.log(` MCP+Search Full ${bar(mcpSearchFullOverhead, maxOvh, 25)} ${padLeft(mcpSearchFullOverhead.toLocaleString(), 6)} tok (2 tools)`); + console.log(` MCP+Search Compact ${bar(mcpSearchCompactOverhead, maxOvh, 25)} ${padLeft(mcpSearchCompactOverhead.toLocaleString(), 6)} tok (3 tools)`); + console.log(` CLI (ocli) ${bar(cliOverhead, maxOvh, 25)} ${padLeft(cliOverhead.toLocaleString(), 6)} tok (1 tool)`); + console.log(); + + // ── Search result size comparison ── + console.log(dline); + console.log(" SEARCH RESULT SIZE — query: \"find pets by status\", top 3"); + console.log(dline); + console.log(); + + const maxRes = mcpSearchFullResultTok; + console.log(` MCP Full search ${bar(mcpSearchFullResultTok, maxRes, 25)} ${padLeft(mcpSearchFullResultTok.toLocaleString(), 6)} tok (name + desc + full JSON schema)`); + console.log(` MCP Compact search ${bar(mcpSearchCompactResultTok, maxRes, 25)} ${padLeft(mcpSearchCompactResultTok.toLocaleString(), 6)} tok (name + desc only)`); + console.log(` CLI search ${bar(cliSearchResultTok, maxRes, 25)} ${padLeft(cliSearchResultTok.toLocaleString(), 6)} tok (name + method + path + desc)`); + console.log(); + console.log(` get_tool_schema ${bar(mcpGetSchemaResultTok, maxRes, 25)} ${padLeft(mcpGetSchemaResultTok.toLocaleString(), 6)} tok (MCP: full schema for 1 tool)`); + console.log(` ocli cmd --help ${bar(cliHelpResultTok, maxRes, 25)} ${padLeft(cliHelpResultTok.toLocaleString(), 6)} tok (CLI: text help for 1 command)`); + console.log(); + + // ── Per-task total cost (realistic multi-turn flows) ── + console.log(dline); + console.log(" TOTAL TOKENS PER TASK (realistic multi-turn agent flow)"); + console.log(dline); + console.log(); + + // MCP Naive: 1 turn = overhead + user msg (20) + assistant output (50) + const mcpNaivePerTask = mcpNaiveOverhead + 20 + 50; + + // MCP+Search Full: 2 turns + // Turn 1: overhead + user(20) + output(30 for search call) + // Turn 2: overhead + user(20) + search_result_full + prev_msgs(50) + output(50 for call_api) + const mcpSearchFullPerTask = + (mcpSearchFullOverhead + 20 + 30) + + (mcpSearchFullOverhead + 20 + mcpSearchFullResultTok + 50 + 50); + + // MCP+Search Compact: 3 turns + // Turn 1: overhead + user(20) + output(30 for search call) + // Turn 2: overhead + user(20) + compact_search_result + prev_msgs(50) + output(30 for get_schema) + // Turn 3: overhead + user(20) + schema_result + prev_msgs(80) + output(50 for call_api) + const mcpSearchCompactPerTask = + (mcpSearchCompactOverhead + 20 + 30) + + (mcpSearchCompactOverhead + 20 + mcpSearchCompactResultTok + 50 + 30) + + (mcpSearchCompactOverhead + 20 + mcpGetSchemaResultTok + 80 + 50); + + // CLI: 3 turns (search → help → execute) + // Turn 1: overhead + user(20) + output(40 for search command) + // Turn 2: overhead + user(20) + search_result + prev_msgs(60) + output(40 for --help command) + // Turn 3: overhead + user(20) + help_result + prev_msgs(100) + output(40 for execute) + const cliPerTask = + (cliOverhead + 20 + 40) + + (cliOverhead + 20 + cliSearchResultTok + 60 + 40) + + (cliOverhead + 20 + cliHelpResultTok + 100 + 40); + + const maxTask = mcpNaivePerTask; + console.log(` MCP Naive ${bar(mcpNaivePerTask, maxTask, 25)} ${padLeft(mcpNaivePerTask.toLocaleString(), 6)} tok (1 turn)`); + console.log(` MCP+Search Full ${bar(mcpSearchFullPerTask, maxTask, 25)} ${padLeft(mcpSearchFullPerTask.toLocaleString(), 6)} tok (2 turns)`); + console.log(` MCP+Search Compact ${bar(mcpSearchCompactPerTask, maxTask, 25)} ${padLeft(mcpSearchCompactPerTask.toLocaleString(), 6)} tok (3 turns)`); + console.log(` CLI (ocli) ${bar(cliPerTask, maxTask, 25)} ${padLeft(cliPerTask.toLocaleString(), 6)} tok (3 turns)`); + console.log(); + + // ── Per-task breakdown ── + console.log(" Per-turn breakdown (MCP+Search Compact vs CLI):"); + console.log(); + console.log(" MCP+Search Compact:"); + console.log(` Turn 1 (search): overhead(${mcpSearchCompactOverhead}) + user(20) + output(30) = ${mcpSearchCompactOverhead + 20 + 30} tok`); + console.log(` Turn 2 (get_schema): overhead(${mcpSearchCompactOverhead}) + user(20) + search_result(${mcpSearchCompactResultTok}) + history(50) + output(30) = ${mcpSearchCompactOverhead + 20 + mcpSearchCompactResultTok + 50 + 30} tok`); + console.log(` Turn 3 (call_api): overhead(${mcpSearchCompactOverhead}) + user(20) + schema(${mcpGetSchemaResultTok}) + history(80) + output(50) = ${mcpSearchCompactOverhead + 20 + mcpGetSchemaResultTok + 80 + 50} tok`); + console.log(); + console.log(" CLI (ocli):"); + console.log(` Turn 1 (search): overhead(${cliOverhead}) + user(20) + output(40) = ${cliOverhead + 20 + 40} tok`); + console.log(` Turn 2 (--help): overhead(${cliOverhead}) + user(20) + search_result(${cliSearchResultTok}) + history(60) + output(40) = ${cliOverhead + 20 + cliSearchResultTok + 60 + 40} tok`); + console.log(` Turn 3 (execute): overhead(${cliOverhead}) + user(20) + help_result(${cliHelpResultTok}) + history(100) + output(40) = ${cliOverhead + 20 + cliHelpResultTok + 100 + 40} tok`); + console.log(); + + // ── 10 tasks total ── + const mcpNaive10 = mcpNaivePerTask * 10; + const mcpSearchFull10 = mcpSearchFullPerTask * 10; + const mcpSearchCompact10 = mcpSearchCompactPerTask * 10; + const cli10 = cliPerTask * 10; + + console.log(` 10 tasks total:`); + console.log(` MCP Naive ${bar(mcpNaive10, mcpNaive10, 25)} ${padLeft(mcpNaive10.toLocaleString(), 7)} tok`); + console.log(` MCP+Search Full ${bar(mcpSearchFull10, mcpNaive10, 25)} ${padLeft(mcpSearchFull10.toLocaleString(), 7)} tok (${mcpSearchFull10 > mcpNaive10 ? "+" : "-"}${Math.abs(((mcpSearchFull10/mcpNaive10 - 1)*100)).toFixed(0)}% vs naive)`); + console.log(` MCP+Search Compact ${bar(mcpSearchCompact10, mcpNaive10, 25)} ${padLeft(mcpSearchCompact10.toLocaleString(), 7)} tok (${mcpSearchCompact10 > mcpNaive10 ? "+" : "-"}${Math.abs(((mcpSearchCompact10/mcpNaive10 - 1)*100)).toFixed(0)}% vs naive)`); + console.log(` CLI (ocli) ${bar(cli10, mcpNaive10, 25)} ${padLeft(cli10.toLocaleString(), 7)} tok (-${((1 - cli10/mcpNaive10)*100).toFixed(0)}% vs naive)`); + console.log(); + + // ── Scaling projection ── + console.log(dline); + console.log(" SCALING: OVERHEAD PER TURN vs ENDPOINT COUNT"); + console.log(dline); + console.log(); + + const tokPerEp = mcpNaiveToolTokens / mcpTools.length; + + const scalePoints = [ + { n: 19, label: "Petstore" }, + { n: 50, label: "" }, + { n: 100, label: "" }, + { n: 200, label: "" }, + { n: 500, label: "" }, + { n: 845, label: "GitHub API" }, + ]; + + console.log(` ${padRight("Endpoints", 11)} ${padRight("MCP Naive", 14)} ${padRight("MCP+S Full", 14)} ${padRight("MCP+S Compact", 16)} ${padRight("CLI (ocli)", 14)} ${padRight("Naive/CLI", 10)}`); + console.log(" " + line); + + for (const pt of scalePoints) { + const naive = Math.ceil(tokPerEp * pt.n) + mcpNaiveSysTok; + const searchFull = mcpSearchFullOverhead; + const searchCompact = mcpSearchCompactOverhead; + const cli = cliOverhead; + const label = pt.label ? ` ← ${pt.label}` : ""; + + console.log( + ` ${padRight(String(pt.n), 11)} ` + + `${padLeft(naive.toLocaleString(), 8)} tok ` + + `${padLeft(searchFull.toLocaleString(), 8)} tok ` + + `${padLeft(searchCompact.toLocaleString(), 10)} tok ` + + `${padLeft(cli.toLocaleString(), 8)} tok ` + + `${padLeft((naive / cli).toFixed(0) + "x", 6)}${label}` + ); + } + console.log(); + console.log(" Note: MCP+Search and CLI overhead is constant regardless of endpoint count."); + console.log(` MCP Naive grows linearly — every endpoint adds ~${Math.round(tokPerEp)} tokens per turn.`); + console.log(); + + // ── Accuracy ── + console.log(dline); + console.log(" BM25 SEARCH ACCURACY (15 natural-language queries)"); + console.log(dline); + console.log(); + + let top1 = 0, top3 = 0, top5 = 0; + const accResults: Array<{ query: string; rank: number; found: string }> = []; + + for (const task of ACCURACY_TASKS) { + const res = searcher.search(task.query, 5); + let rank = 0; + let found = "(miss)"; + for (let i = 0; i < res.length; i++) { + if (matches(res[i].name, task.expected) || matches(res[i].path, task.expectedPath)) { + rank = i + 1; + found = res[i].name; + break; + } + } + if (rank === 1) { top1++; top3++; top5++; } + else if (rank <= 3) { top3++; top5++; } + else if (rank <= 5) { top5++; } + accResults.push({ query: task.query, rank, found }); + } + + const total = ACCURACY_TASKS.length; + + for (const r of accResults) { + const icon = r.rank === 1 ? "✓" : r.rank <= 3 ? "~" : r.rank <= 5 ? "·" : "✗"; + const rankStr = r.rank > 0 ? `#${r.rank}` : "miss"; + console.log(` ${icon} ${padRight(rankStr, 5)} ${padRight(`"${r.query}"`, 42)} ${r.found}`); + } + + console.log(); + console.log(` Top-1: ${top1}/${total} (${((top1/total)*100).toFixed(0)}%) Top-3: ${top3}/${total} (${((top3/total)*100).toFixed(0)}%) Top-5: ${top5}/${total} (${((top5/total)*100).toFixed(0)}%)`); + console.log(); + console.log(` All strategies use the same BM25 engine — accuracy is identical.`); + console.log(); + + // ── Monthly cost at scale ── + console.log(dline); + console.log(" MONTHLY COST ESTIMATE (100 tasks/day, Claude Sonnet $3/M input)"); + console.log(dline); + console.log(); + + const price = 3.0; + const dailyTasks = 100; + + const costLine = (label: string, tokPerTask: number) => { + const monthly = (tokPerTask * dailyTasks * 30 / 1_000_000) * price; + return ` ${padRight(label, 22)} ${padLeft(tokPerTask.toLocaleString(), 7)} tok/task $${padLeft(monthly.toFixed(2), 8)}/month`; + }; + + console.log(` ${padRight("", 22)} ${padRight("Per task", 18)} Monthly cost`); + console.log(" " + line); + console.log(` 19 endpoints (Petstore):`); + console.log(costLine(" MCP Naive", mcpNaivePerTask)); + console.log(costLine(" MCP+Search Full", mcpSearchFullPerTask)); + console.log(costLine(" MCP+Search Compact", mcpSearchCompactPerTask)); + console.log(costLine(" CLI (ocli)", cliPerTask)); + console.log(); + + // At 845 endpoints + const naive845overhead = Math.ceil(tokPerEp * 845) + mcpNaiveSysTok; + const naive845task = naive845overhead + 20 + 50; + // For 845-endpoint API, search results are bigger (more complex schemas) + const scaleFactor845 = 845 / 19; // schemas scale with API complexity + const searchFull845resultTok = Math.ceil(mcpSearchFullResultTok * Math.sqrt(scaleFactor845)); + const searchFull845task = (mcpSearchFullOverhead + 20 + 30) + (mcpSearchFullOverhead + 20 + searchFull845resultTok + 50 + 50); + const getSchema845resultTok = Math.ceil(mcpGetSchemaResultTok * Math.sqrt(scaleFactor845)); + const searchCompact845task = + (mcpSearchCompactOverhead + 20 + 30) + + (mcpSearchCompactOverhead + 20 + mcpSearchCompactResultTok + 50 + 30) + + (mcpSearchCompactOverhead + 20 + getSchema845resultTok + 80 + 50); + const cliHelp845resultTok = Math.ceil(cliHelpResultTok * Math.sqrt(scaleFactor845)); + const cli845task = + (cliOverhead + 20 + 40) + + (cliOverhead + 20 + cliSearchResultTok + 60 + 40) + + (cliOverhead + 20 + cliHelp845resultTok + 100 + 40); + + console.log(` 845 endpoints (GitHub API scale):`); + console.log(costLine(" MCP Naive", naive845task)); + console.log(costLine(" MCP+Search Full", searchFull845task)); + console.log(costLine(" MCP+Search Compact", searchCompact845task)); + console.log(costLine(" CLI (ocli)", cli845task)); + console.log(); + + // ── Final verdict ── + console.log(dline); + console.log(" VERDICT"); + console.log(dline); + console.log(); + console.log(` ${padRight("", 22)} ${padRight("Overhead/turn", 16)} ${padRight("Turns", 8)} ${padRight("Search result", 16)} ${padRight("Accuracy", 10)} Server?`); + console.log(" " + line); + console.log(` ${padRight("MCP Naive", 22)} ${padLeft(mcpNaiveOverhead.toLocaleString(), 6)} tok ${padRight("1", 8)} ${padRight("N/A", 16)} ${padRight("100%", 10)} Yes`); + console.log(` ${padRight("MCP+Search Full", 22)} ${padLeft(mcpSearchFullOverhead.toLocaleString(), 6)} tok ${padRight("2", 8)} ${padLeft(mcpSearchFullResultTok.toLocaleString(), 5)} tok/query ${padLeft(((top3/total)*100).toFixed(0) + "%", 5)}${" ".repeat(5)} Yes`); + console.log(` ${padRight("MCP+Search Compact", 22)} ${padLeft(mcpSearchCompactOverhead.toLocaleString(), 6)} tok ${padRight("3", 8)} ${padLeft(mcpSearchCompactResultTok.toLocaleString(), 5)} tok/query ${padLeft(((top3/total)*100).toFixed(0) + "%", 5)}${" ".repeat(5)} Yes`); + console.log(` ${padRight("CLI (ocli)", 22)} ${padLeft(cliOverhead.toLocaleString(), 6)} tok ${padRight("3", 8)} ${padLeft(cliSearchResultTok.toLocaleString(), 5)} tok/query ${padLeft(((top3/total)*100).toFixed(0) + "%", 5)}${" ".repeat(5)} No`); + console.log(); + console.log(" Key insights:"); + console.log(" - MCP Naive is simplest but scales terribly (130K+ tok at 845 endpoints)"); + console.log(" - MCP+Search Full has low overhead but search results carry full JSON schemas"); + console.log(" - MCP+Search Compact is the fairest MCP comparison to CLI (same flow: search → schema → call)"); + console.log(` - CLI and MCP Compact have similar search results; CLI wins on overhead (${cliOverhead} vs ${mcpSearchCompactOverhead} tok/turn)`); + console.log(" - CLI needs no MCP server — any agent with shell access works"); + console.log(" - All search strategies share the same BM25 accuracy"); + console.log(); +} + +main().catch((err) => { + console.error(err); + process.exit(1); +}); diff --git a/skills/ocli-api/ISSUE.md b/skills/ocli-api/ISSUE.md new file mode 100644 index 0000000..f7d7e1f --- /dev/null +++ b/skills/ocli-api/ISSUE.md @@ -0,0 +1,90 @@ +# Issue draft for openclaw/openclaw + +**Title:** Skill: ocli-api — call any REST API via CLI, no MCP server needed + +**Body:** + +## Problem + +Connecting AI agents to REST APIs today means either: + +1. **MCP Naive** — register every endpoint as a tool. At 100+ endpoints, that's 15K+ tokens of JSON schemas injected every turn. At GitHub API scale (845 endpoints) it's 130K tokens/turn. +2. **MCP+Search** — better, but still requires a running MCP server, transport layer, and search results carry full JSON schemas back to context. + +Both approaches require a persistent server process and MCP-aware client. This is the "MCP madness" — every new API means another server to run, another transport to manage, another set of tool schemas eating your context window. + +## Solution + +[openapi-to-cli](https://github.com/EvilFreelancer/openapi-to-cli) (`ocli`) turns any OpenAPI/Swagger spec into CLI commands at runtime. The agent needs only 1 tool (`execute_command`) and discovers endpoints via BM25 search: + +``` +Agent: ocli commands --query "create pull request" --limit 3 +→ repos_owner_repo_pulls_post post /repos/{owner}/{repo}/pulls Create a pull request + +Agent: ocli repos_owner_repo_pulls_post --help +→ Options: --owner (required) --repo (required) --title --head --base --body + +Agent: ocli repos_owner_repo_pulls_post --owner octocat --repo hello --title "Fix bug" --head feature --base main +→ { "number": 42, "html_url": "..." } +``` + +**Benchmark results (honest, all strategies use same BM25 engine):** + +| Strategy | Overhead/turn | Turns | Search result | Server? | +|----------|--------------|-------|---------------|---------| +| MCP Naive (19 ep) | 2,945 tok | 1 | N/A | Yes | +| MCP+Search Full | 355 tok | 2 | 1,305 tok/query | Yes | +| MCP+Search Compact | 437 tok | 3 | 61 tok/query | Yes | +| **CLI (ocli)** | **158 tok** | **3** | **67 tok/query** | **No** | + +At 845 endpoints, MCP Naive costs $1,172/month vs CLI at $11/month (100 tasks/day, Sonnet $3/M). + +## Proposed skill + +```yaml +--- +name: ocli-api +description: Turn any OpenAPI/Swagger API into CLI commands and call them. Search endpoints with BM25, check parameters, execute — no MCP server needed. +version: 1.0.0 +user-invocable: true +metadata: {"openclaw":{"emoji":"🔌","requires":{"bins":["ocli"]}}} +homepage: https://github.com/EvilFreelancer/openapi-to-cli +--- +``` + +The full `SKILL.md` is available at: https://github.com/EvilFreelancer/openapi-to-cli/tree/main/skills/ocli-api + +### What the skill teaches the agent + +1. **Search** — `ocli commands --query "..." --limit 5` (BM25 ranked) +2. **Inspect** — `ocli --help` (text parameters, not JSON schemas) +3. **Execute** — `ocli --flag value` (plain HTTP, JSON response) + +### Installation + +Once the skill is published to ClawHub: + +```bash +# Install the skill +clawhub install ocli-api + +# Prerequisite: install ocli itself +npm install -g openapi-to-cli + +# Onboard your first API +ocli profiles add myapi \ + --api-base-url https://api.example.com \ + --openapi-spec https://api.example.com/openapi.json \ + --api-bearer-token "$TOKEN" +``` + +Or manually — copy `skills/ocli-api/SKILL.md` to `~/.openclaw/skills/ocli-api/SKILL.md`. + +### Why this belongs in openclaw skills + +- **Zero infrastructure** — `npm install -g openapi-to-cli`, no server process +- **Universal** — works with any OpenAPI/Swagger spec (Bitrix24, GitHub, Box, Petstore, internal APIs) +- **Token-efficient** — 158 tok overhead vs 2,945+ for MCP approaches +- **Composable** — pipes, chains, shell scripts natively +- **Multiple APIs** — profiles with per-API auth, endpoint filtering, command prefixes +- **Replaces MCP for REST APIs** — agent doesn't need MCP-aware client, just shell access diff --git a/skills/ocli-api/SKILL.md b/skills/ocli-api/SKILL.md new file mode 100644 index 0000000..ebb062d --- /dev/null +++ b/skills/ocli-api/SKILL.md @@ -0,0 +1,86 @@ +--- +name: ocli-api +description: Turn any OpenAPI/Swagger API into CLI commands and call them. Search endpoints with BM25, check parameters, execute — no MCP server needed. +version: 1.0.0 +user-invocable: true +disable-model-invocation: false +metadata: {"openclaw":{"emoji":"🔌","requires":{"bins":["ocli"],"env":[]}}} +homepage: https://github.com/EvilFreelancer/openapi-to-cli +--- + +# ocli — OpenAPI to CLI + +Call any HTTP API described by an OpenAPI/Swagger spec as CLI commands. +No MCP server, no code generation, no JSON schemas in context. + +## When to use + +- You need to call a REST API (internal, cloud, SaaS) +- You have an OpenAPI or Swagger spec (URL or local file) +- You want minimal token overhead (1 tool, ~158 tokens/turn) + +## Setup (one-time) + +```bash +npm install -g openapi-to-cli + +ocli profiles add \ + --api-base-url \ + --openapi-spec \ + --api-bearer-token "$TOKEN" + +ocli use +``` + +## Workflow + +1. **Search** for the right command: + ```bash + ocli commands --query "your task description" --limit 5 + ``` +2. **Check parameters** of the chosen command: + ```bash + ocli --help + ``` +3. **Execute** the command: + ```bash + ocli --param1 value1 --param2 value2 + ``` +4. **Parse** the JSON response and act on the result. + +## Search options + +```bash +# BM25 natural language search +ocli commands --query "upload file to storage" --limit 5 + +# Regex pattern search +ocli commands --regex "users.*post" --limit 10 + +# List all commands +ocli commands +``` + +## Multiple APIs + +```bash +# Switch active profile +ocli use github + +# Or specify per-call +ocli repos_get --profile github --owner octocat --repo Hello-World +``` + +## Guardrails + +- Always search before guessing a command name. +- Always check `--help` before calling a command you haven't used before. +- Never fabricate parameter names — use the ones from `--help` output. +- If a command returns an error, read the response body before retrying. + +## Failure handling + +- **Command not found**: re-search with different keywords or use `--regex`. +- **Missing required parameter**: run `--help` and add the missing flag. +- **401/403**: check that the profile has a valid token (`ocli profiles show `). +- **Spec not loaded**: run `ocli profiles add` again with `--openapi-spec` to refresh cache. From 72f94dcbb98c1f7af5716df514dd3346509df1f3 Mon Sep 17 00:00:00 2001 From: "v.kovalskii" Date: Tue, 17 Mar 2026 17:04:28 +0300 Subject: [PATCH 2/3] Add 4-layer positioning: Built-in Tools, MCP, Skills, CLI Co-Authored-By: Claude Opus 4.6 (1M context) --- README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 03a7bd0..404f93d 100644 --- a/README.md +++ b/README.md @@ -14,15 +14,16 @@ ocli commands --query "create pull request" --limit 3 ocli repos_owner_repo_pulls_post --owner octocat --repo hello --title "Fix bug" --head feature --base main ``` -### Where CLI fits: MCP, Skills, and CLI +### Where CLI fits: Tools, MCP, Skills, and CLI -MCP, skills, and CLI are not competing approaches — they solve different problems at different layers: +Tools, MCP, skills, and CLI are not competing approaches — they solve different problems at different layers: -| Layer | Tool | Best for | +| Layer | What | Best for | |-------|------|----------| -| **Infrastructure** | MCP | Centralized APIs, enterprise SSO, shared state between agents, persistent connections | -| **Agent knowledge** | Skills | On-demand instructions, context isolation, teaching agents _when_ to use a tool | -| **Runtime execution** | CLI | Long action chains, automation, shell pipelines, minimal per-turn token cost | +| **Built-in tools** | Standard agent toolset | Critical capabilities that must always be in context (file read/write, shell, browser) | +| **MCP** | Remote tool servers | APIs that need centralized auth, enterprise SSO, shared state, persistent connections, or can't be in standard delivery | +| **Skills** | On-demand instructions | Context isolation, teaching agents _when_ and _how_ to use a tool — loaded only when needed | +| **CLI** | Runtime execution | Long action chains, automation, shell pipelines — agent already knows what to do | `ocli` lives at the **runtime layer**. When an agent needs to call a REST API — search for the right endpoint, check its parameters, execute the call — CLI does this with minimal context overhead and zero infrastructure. From 79703cc7e07de4efe2de6263c172ff3667d6bcca Mon Sep 17 00:00:00 2001 From: "v.kovalskii" Date: Tue, 17 Mar 2026 17:21:23 +0300 Subject: [PATCH 3/3] Add CHANGELOG.md Closes #7 Co-Authored-By: Claude Opus 4.6 (1M context) --- CHANGELOG.md | 63 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..f16c30f --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,63 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +### Added +- OpenClaw skill (`skills/ocli-api/SKILL.md`) published to [ClawHub](https://clawhub.ai/skills/ocli-api) +- Fair 4-strategy token benchmark: MCP Naive, MCP+Search Full, MCP+Search Compact, CLI +- mcp2cli added to comparison table with MCP/GraphQL/TOON/OAuth features +- CHANGELOG.md + +### Changed +- README rewritten: "CLI vs MCP" positioning replaced with 4-layer model (Built-in Tools, MCP, Skills, CLI) +- README reduced from 285 to 175 lines — removed implementation details available via `--help` +- Benchmark now uses same BM25 engine for all strategies (fair comparison) + +### Fixed +- Nested JSON values for body flags now parsed correctly (#5, thanks @veged) + +## [0.1.3] - 2026-03-12 + +### Added +- BM25 command search (`ocli commands --query "..."`) +- Regex command search (`ocli commands --regex "..."`) +- YAML spec support (Box API 258 endpoints tested) +- GitHub API test fixture (845 endpoints) +- Box API test fixture (258 endpoints) +- `ocli commands` replaces deprecated `ocli search` +- CLI-Anything added to comparison table +- MIT license file + +## [0.1.2] - 2026-03-12 + +### Added +- Command generation from OpenAPI paths and methods + +## [0.1.1] - 2026-03-12 + +### Fixed +- Version tag generation + +## [0.1.0] - 2026-03-12 + +### Added +- Initial release +- OpenAPI/Swagger spec loading (URL and local file) +- Spec caching in `.ocli/specs/` +- Profile management (`profiles add/list/show/remove`, `use`) +- Command generation from OpenAPI paths with method suffix logic +- Path and query parameter extraction +- HTTP request execution (GET, POST, PUT, DELETE, PATCH) +- Basic and Bearer token authentication +- GitHub Actions CI/CD workflows + +[Unreleased]: https://github.com/EvilFreelancer/openapi-to-cli/compare/v0.1.3...HEAD +[0.1.3]: https://github.com/EvilFreelancer/openapi-to-cli/compare/v0.1.2...v0.1.3 +[0.1.2]: https://github.com/EvilFreelancer/openapi-to-cli/compare/v0.1.1...v0.1.2 +[0.1.1]: https://github.com/EvilFreelancer/openapi-to-cli/compare/v0.1.0...v0.1.1 +[0.1.0]: https://github.com/EvilFreelancer/openapi-to-cli/releases/tag/v0.1.0