[DX-793] [DX-938] Standardize JSON output with envelope format, fail() error handling, and format helpers

.claude/skills/ably-new-command/SKILL.md

@@ -172,9 +172,14 @@ if (!this.shouldOutputJson(flags)) {
   this.log(formatListening("Listening for messages."));
 }
 
-// JSON output:
+// JSON output — use logJsonResult for one-shot results:
 if (this.shouldOutputJson(flags)) {
-  this.log(this.formatJsonOutput(data, flags));
+  this.logJsonResult({ channel: args.channel, message }, flags);
+}
+
+// Streaming events — use logJsonEvent:
+if (this.shouldOutputJson(flags)) {
+  this.logJsonEvent({ eventType: event.type, message, channel: channelName }, flags);
 }
 ```
 
@@ -208,38 +213,84 @@ Rules:
 - `formatLabel(text)` — dim with colon, for field labels
 - `formatHeading(text)` — bold, for record headings in lists
 - `formatIndex(n)` — dim bracketed number, for history ordering
-- Use `this.handleCommandError()` for all errors (see Error handling below), never `this.log(chalk.red(...))`
+- Use `this.fail()` for all errors (see Error handling below), never `this.log(chalk.red(...))`
 - Never use `console.log` or `console.error` — always `this.log()` or `this.logToStderr()`
 
+### JSON envelope — reserved keys
+
+`logJsonResult(data, flags)` and `logJsonEvent(data, flags)` are shorthand for `this.log(this.formatJsonRecord("result"|"event", data, flags))`. The envelope wraps data in `{type, command, success?, ...data}` and **silently strips** these reserved keys from your data to prevent collisions:
+- `type` — always stripped (envelope's own `type` field)
+- `command` — always stripped (envelope's own `command` field)
+- `success` — stripped from `"error"` records (always `false`); for `"result"` records, data's `success` **overrides** the envelope's default `true`
+
+**Pitfall:** If your event data has a `type` field (e.g., from an SDK event object), it will be silently dropped. Use a different key name:
+```typescript
+// WRONG — event.type is silently stripped by the envelope
+this.logJsonEvent({ type: event.type, message, room }, flags);
+
+// CORRECT — use "eventType" to avoid collision with envelope's "type"
+this.logJsonEvent({ eventType: event.type, message, room }, flags);
+```
+
+Similarly, for batch results with a success/failure summary, don't use `success` as the key — it collides with the envelope's `success: true`:
+```typescript
+// WRONG — data's "success" overrides envelope's "success"
+this.logJsonResult({ success: errors === 0, published, errors }, flags);
+
+// CORRECT — use "allSucceeded" for the batch summary
+this.logJsonResult({ allSucceeded: errors === 0, published, errors }, flags);
+```
+
 ### Error handling
 
-**Use `handleCommandError` for all errors.** It's the single error function for commands — it logs the CLI event, emits JSON error when `--json` is active, and calls `this.error()` for human-readable output. It accepts an `Error` object or a plain string message.
+Choose the right mechanism based on intent:
+
+| Intent | Method | Behavior |
+|--------|--------|----------|
+| **Stop the command** (fatal error) | `this.fail(error, flags, component)` | Logs event, emits JSON error envelope if `--json`, exits. Returns `never` — execution stops, no `return;` needed. |
+| **Warn and continue** (non-fatal) | `this.warn()` or `this.logToStderr()` | Prints warning, execution continues normally. |
+| **Reject inside Promise callbacks** | `reject(new Error(...))` | Propagates to `await`, where the catch block calls `this.fail()`. |
+
+All fatal errors flow through `this.fail()`, which uses `CommandError` (`src/errors/command-error.ts`) to preserve Ably error codes and HTTP status codes:
+
+```
+this.fail(): never   ← the single funnel (logs event, emits JSON, exits)
+    ↓ internally calls
+this.error()         ← oclif exit (ONLY inside fail, nowhere else)
+```
+
+**In command `run()` methods**, use `this.fail()` for all errors. It always exits — returns `never`, so no `return;` is needed after calling it. It logs the CLI event, preserves structured error data, emits JSON error envelope when `--json` is active, and calls `this.error()` for human-readable output. It accepts an `Error` object or a plain string message.
 
 ```typescript
 // In catch blocks — pass the error object
 try {
-  // command logic
+  // All fallible calls go inside try-catch, including base class methods
+  // like createControlApi, createAblyRealtimeClient, etc.
+  const controlApi = this.createControlApi(flags);
+  const result = await controlApi.someMethod(appId, data);
+  // ...
 } catch (error) {
-  this.handleCommandError(
+  this.fail(
     error,
     flags,
     "ComponentName",     // e.g., "ChannelPublish", "PresenceEnter"
     { channel: args.channel },  // optional context for logging
   );
 }
 
-// For validation / early exit — pass a string message
+// For validation / early exit — pass a string message (no return; needed)
 if (!appId) {
-  this.handleCommandError(
+  this.fail(
     'No app specified. Use --app flag or select an app with "ably apps switch"',
     flags,
     "AppResolve",
   );
-  return;
 }
 ```
 
-**Do NOT use `this.error()` or `this.jsonError()` directly** — they are internal implementation details. Calling `this.error()` directly skips event logging and doesn't respect `--json` mode. Calling `this.jsonError()` directly skips event logging and doesn't handle the non-JSON case.
+**In base class utility methods** (e.g., `createControlApi`, `createAblyRealtimeClient`, `parseJsonFlag`), use `throw new Error()`. These methods return values, so they can't call `fail`. The thrown error is caught by the command's try-catch and routed through `fail`.
+
+**Do NOT use `this.error()` directly** — it is an internal implementation detail of `fail`. Calling `this.error()` directly skips event logging and doesn't respect `--json` mode.
 
 ### Pattern-specific implementation
 
@@ -316,7 +367,10 @@ pnpm test:unit      # Run tests
 - [ ] Output helpers used correctly (`formatProgress`, `formatSuccess`, `formatListening`, `formatResource`, `formatTimestamp`, `formatClientId`, `formatEventType`, `formatLabel`, `formatHeading`, `formatIndex`)
 - [ ] `success()` messages end with `.` (period)
 - [ ] Resource names use `resource(name)`, never quoted
-- [ ] Error handling uses `this.handleCommandError()` exclusively, not `this.error()`, `this.jsonError()`, or `this.log(chalk.red(...))`
+- [ ] JSON output uses `logJsonResult()` (one-shot) or `logJsonEvent()` (streaming), not direct `formatJsonRecord()`
+- [ ] Subscribe/enter commands use `this.waitAndTrackCleanup(flags, component, flags.duration)` (not `waitUntilInterruptedOrTimeout`)
+- [ ] Error handling uses `this.fail()` exclusively, not `this.error()` or `this.log(chalk.red(...))`
+- [ ] At least one `--json` example in `static examples`
 - [ ] Test file at matching path under `test/unit/commands/`
 - [ ] Tests use correct mock helper (`getMockAblyRealtime`, `getMockAblyRest`, `nock`)
 - [ ] Tests don't pass auth flags — `MockConfigManager` handles auth

.claude/skills/ably-new-command/references/patterns.md

@@ -6,6 +6,7 @@ Pick the pattern that matches your command from Step 1 of the skill, then follow
 - [Subscribe Pattern](#subscribe-pattern)
 - [Publish/Send Pattern](#publishsend-pattern)
 - [History Pattern](#history-pattern)
+- [Get Pattern](#get-pattern)
 - [Enter/Presence Pattern](#enterpresence-pattern)
 - [List Pattern](#list-pattern)
 - [CRUD / Control API Pattern](#crud--control-api-pattern)
@@ -58,18 +59,24 @@ async run(): Promise<void> {
     sequenceCounter++;
     // Format and output the message
     if (this.shouldOutputJson(flags)) {
-      this.log(this.formatJsonOutput({ /* message data */ }, flags));
+      // Use "event" type for streaming records. IMPORTANT: don't use "type" as a
+      // data key — it's reserved by the envelope. Use "eventType" instead.
+      this.logJsonEvent({
+        eventType: "message",  // not "type" — that's reserved by the envelope
+        channel: args.channel,
+        data: message.data,
+        name: message.name,
+        timestamp: message.timestamp,
+      }, flags);
     } else {
       // Human-readable output with formatTimestamp, formatResource, chalk colors
     }
   });
 
-  await waitUntilInterruptedOrTimeout(flags);
+  await this.waitAndTrackCleanup(flags, "MySubscribe", flags.duration);
 }
 ```
 
-Import `waitUntilInterruptedOrTimeout` from `../../utils/long-running.js`.
-
 ---
 
 ## Publish/Send Pattern
@@ -105,12 +112,14 @@ async run(): Promise<void> {
     await channel.publish(message as Ably.Message);
 
     if (this.shouldOutputJson(flags)) {
-      this.log(this.formatJsonOutput({ success: true, channel: args.channel }, flags));
+      // Use "result" type for one-shot results. Don't use "success" as a data key
+      // for batch summaries — it overrides the envelope's success field. Use "allSucceeded".
+      this.logJsonResult({ channel: args.channel }, flags);
     } else {
       this.log(formatSuccess("Message published to channel: " + formatResource(args.channel) + "."));
     }
   } catch (error) {
-    this.handleCommandError(error, flags, "Publish", { channel: args.channel });
+    this.fail(error, flags, "Publish", { channel: args.channel });
   }
 }
 ```
@@ -148,7 +157,7 @@ async run(): Promise<void> {
   const messages = history.items;
 
   if (this.shouldOutputJson(flags)) {
-    this.log(this.formatJsonOutput({ messages }, flags));
+    this.logJsonResult({ messages }, flags);
   } else {
     this.log(formatSuccess(`Found ${messages.length} messages.`));
     // Display each message
@@ -158,6 +167,44 @@ async run(): Promise<void> {
 
 ---
 
+## Get Pattern
+
+Get commands perform one-shot queries for current state. They use REST clients and don't need `clientIdFlag`, `durationFlag`, or `rewindFlag`.
+
+```typescript
+static override flags = {
+  ...productApiFlags,
+  // command-specific flags here
+};
+```
+
+```typescript
+async run(): Promise<void> {
+  const { args, flags } = await this.parse(MyGetCommand);
+
+  try {
+    const client = await this.createAblyRestClient(flags);
+    if (!client) return;
+
+    // Fetch the resource data
+    const result = await client.request("get", `/resource/${encodeURIComponent(args.id)}`, 2);
+    const data = result.items?.[0] || {};
+
+    if (this.shouldOutputJson(flags)) {
+      this.logJsonResult({ resource: args.id, ...data }, flags);
+    } else {
+      this.log(`Details for ${formatResource(args.id)}:\n`);
+      this.log(`${formatLabel("Field")} ${data.field}`);
+      this.log(`${formatLabel("Status")} ${data.status}`);
+    }
+  } catch (error) {
+    this.fail(error, flags, "ResourceGet", { resource: args.id });
+  }
+}
+```
+
+---
+
 ## Enter/Presence Pattern
 
 Flags for enter commands:
@@ -189,8 +236,7 @@ async run(): Promise<void> {
     try {
       presenceData = JSON.parse(flags.data);
     } catch {
-      this.handleCommandError("Invalid JSON data provided", flags, "PresenceEnter");
-      return;
+      this.fail("Invalid JSON data provided", flags, "PresenceEnter");
     }
   }
 
@@ -213,7 +259,7 @@ async run(): Promise<void> {
     this.log(formatListening("Present on channel."));
   }
 
-  await waitUntilInterruptedOrTimeout(flags);
+  await this.waitAndTrackCleanup(flags, "PresenceEnter", flags.duration);
 }
 
 // Clean up in finally — leave presence before closing connection
@@ -260,24 +306,23 @@ Full Control API list command template:
 async run(): Promise<void> {
   const { flags } = await this.parse(MyListCommand);
 
-  const controlApi = this.createControlApi(flags);
   const appId = await this.resolveAppId(flags);
 
   if (!appId) {
-    this.handleCommandError(
+    this.fail(
       'No app specified. Use --app flag or select an app with "ably apps switch"',
       flags,
       "ListItems",
     );
-    return;
   }
 
   try {
+    const controlApi = this.createControlApi(flags);
     const items = await controlApi.listThings(appId);
     const limited = flags.limit ? items.slice(0, flags.limit) : items;
 
     if (this.shouldOutputJson(flags)) {
-      this.log(this.formatJsonOutput({ items: limited, total: limited.length, appId }, flags));
+      this.logJsonResult({ items: limited, total: limited.length, appId }, flags);
     } else {
       this.log(`Found ${limited.length} item${limited.length !== 1 ? "s" : ""}:\n`);
       for (const item of limited) {
@@ -288,7 +333,7 @@ async run(): Promise<void> {
       }
     }
   } catch (error) {
-    this.handleCommandError(error, flags, "ListItems");
+    this.fail(error, flags, "ListItems");
   }
 }
 ```
@@ -307,29 +352,28 @@ Key conventions for list output:
 async run(): Promise<void> {
   const { args, flags } = await this.parse(MyControlCommand);
 
-  const controlApi = this.createControlApi(flags);
   const appId = await this.resolveAppId(flags);
 
   if (!appId) {
-    this.handleCommandError(
+    this.fail(
       'No app specified. Use --app flag or select an app with "ably apps switch"',
       flags,
       "CreateResource",
     );
-    return;
   }
 
   try {
+    const controlApi = this.createControlApi(flags);
     const result = await controlApi.someMethod(appId, data);
 
     if (this.shouldOutputJson(flags)) {
-      this.log(this.formatJsonOutput({ result }, flags));
+      this.logJsonResult({ resource: result }, flags);
     } else {
       this.log(formatSuccess("Resource created: " + formatResource(result.id) + "."));
       // Display additional fields
     }
   } catch (error) {
-    this.handleCommandError(error, flags, "CreateResource");
+    this.fail(error, flags, "CreateResource");
   }
 }
 ```

AGENTS.md

@@ -197,12 +197,38 @@ All output helpers use the `format` prefix and are exported from `src/utils/outp
 - **Count labels**: `formatCountLabel(n, "message")` — cyan count + pluralized label.
 - **Limit warnings**: `formatLimitWarning(count, limit, "items")` — yellow warning if results truncated.
 - **JSON guard**: All human-readable output (progress, success, listening messages) must be wrapped in `if (!this.shouldOutputJson(flags))` so it doesn't pollute `--json` output. Only JSON payloads should be emitted when `--json` is active.
-- **JSON errors**: In catch blocks, use `this.handleCommandError(error, flags, component, context?)` for consistent error handling. It logs the event, emits JSON error when `--json` is active, and calls `this.error()` for human-readable output. For non-standard error flows, use `this.jsonError()` directly.
+- **JSON envelope**: Use `this.logJsonResult(data, flags)` for one-shot results and `this.logJsonEvent(data, flags)` for streaming events. These are shorthand for `this.log(this.formatJsonRecord("result"|"event", data, flags))`. The envelope wraps data in `{type, command, success?, ...data}`. Do NOT add ad-hoc `success: true/false` — the envelope handles it. `--json` produces compact single-line output (NDJSON for streaming). `--pretty-json` is unchanged.
+- **JSON errors**: Use `this.fail(error, flags, component, context?)` as the single error funnel in command `run()` methods. It logs the CLI event, preserves structured error data (Ably codes, HTTP status), emits JSON error envelope when `--json` is active, and calls `this.error()` for human-readable output. Returns `never` — no `return;` needed after calling it. Do NOT call `this.error()` directly — it is an internal implementation detail of `fail`.
 - **History output**: Use `[index] timestamp` ordering: `` `${formatIndex(index + 1)} ${formatTimestamp(timestamp)}` ``. Consistent across all history commands (channels, logs, connection-lifecycle, push).
 
+### Error handling architecture
+
+Choose the right mechanism based on intent:
+
+| Intent | Method | Behavior |
+|--------|--------|----------|
+| **Stop the command** (fatal error) | `this.fail(error, flags, component)` | Logs event, emits JSON error envelope if `--json`, exits. Returns `never` — execution stops, no `return;` needed. |
+| **Warn and continue** (non-fatal) | `this.warn()` or `this.logToStderr()` | Prints warning, execution continues normally. |
+| **Reject inside Promise callbacks** | `reject(new Error(...))` | Propagates to `await`, where the catch block calls `this.fail()`. |
+
+All fatal errors flow through `this.fail()` (`src/base-command.ts`), which uses `CommandError` (`src/errors/command-error.ts`) to preserve Ably error codes and HTTP status codes:
+
+```
+this.fail(): never   ← the single funnel (logs event, emits JSON, exits)
+    ↓ internally calls
+this.error()         ← oclif exit (ONLY inside fail, nowhere else)
+```
+
+- **`this.fail()` always exits** — it returns `never`. TypeScript enforces no code runs after it. This eliminates the "forgotten `return;`" bug class.
+- **In command `run()` methods**: Use `this.fail()` for all errors. Wrap fallible calls in try-catch blocks.
+- **Base class methods with `flags`** (`createControlApi`, `createAblyRealtimeClient`, `requireAppId`, `runControlCommand`, etc.) also use `this.fail()` directly. Methods without `flags` pass `{}` as a fallback.
+- **`reject(new Error(...))`** inside Promise callbacks (e.g., connection event handlers) is the one pattern that can't use `this.fail()` — the rejection propagates to `await`, where the command's catch block calls `this.fail()`.
+- **Never use `this.error()` directly** — it is an internal implementation detail of `this.fail()`.
+- **`requireAppId`** returns `Promise<string>` (not nullable) — calls `this.fail()` internally if no app found.
+- **`runControlCommand<T>`** returns `Promise<T>` (not nullable) — calls `this.fail()` internally on error.
+
 ### Additional output patterns (direct chalk, not helpers)
 - **Warnings**: `chalk.yellow("Warning: ...")` — for non-fatal warnings
-- **Errors**: Use `this.error()` (oclif standard) for fatal errors, not `this.log(chalk.red(...))`
 - **No app error**: `'No app specified. Use --app flag or select an app with "ably apps switch"'`
 
 ### Help output theme

README.md

@@ -4103,6 +4103,19 @@ EXAMPLES
 _See code: [src/commands/support/contact.ts](https://github.com/ably/ably-cli/blob/v0.17.0/src/commands/support/contact.ts)_
 <!-- commandsstop -->
 
+## JSON Output
+
+All commands support `--json` for machine-readable output and `--pretty-json` for human-readable formatted JSON.
+
+When using `--json`, every record is wrapped in a standard envelope:
+
+- **`type`** — `"result"`, `"event"`, `"error"`, or `"log"`
+- **`command`** — the command that produced the record (e.g. `"channels:publish"`)
+- **`success`** — `true` or `false` (only on `"result"` and `"error"` types)
+- Additional fields are command-specific
+
+Streaming commands (subscribe, logs) emit one JSON object per line (NDJSON).
+
 ## Environment Variables
 
 The CLI supports the following environment variables for authentication and configuration:

docs/Project-Structure.md

@@ -105,7 +105,8 @@ This document outlines the directory structure of the Ably CLI project.
 │   │   ├── mock-ably-spaces.ts     # Mock Ably Spaces SDK
 │   │   ├── mock-config-manager.ts  # MockConfigManager (provides test auth)
 │   │   ├── mock-control-api-keys.ts # Mock Control API key responses
-│   │   └── ably-event-emitter.ts   # Event emitter helper for mock SDKs
+│   │   ├── ably-event-emitter.ts   # Event emitter helper for mock SDKs
+│   │   └── ndjson.ts                # NDJSON parsing helpers (parseNdjsonLines, parseLogLines, captureJsonLogs)
 │   ├── unit/                   # Fast, mocked tests
 │   │   ├── setup.ts            # Unit test setup
 │   │   ├── base/               # Base command class tests