Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
5e30914
Add ably-common submodule
paddybyers Mar 30, 2026
bada341
Add UTS (Universal Test Spec) suite for REST API
paddybyers Apr 30, 2026
7bcbd92
Add UTS specs for Realtime connection lifecycle
paddybyers Mar 30, 2026
aa17d10
Fix connection test specs to close WebSocket transport when necessary
paddybyers Mar 30, 2026
0cf678c
Refactor realtime test specs: extract mock WebSocket helper and add s…
paddybyers Mar 30, 2026
e914b8d
Refactor test specs to use EXPECT_THROW instead of TRY/CATCH
paddybyers Mar 30, 2026
f7e7a6e
Use unique channel names in test specs to prevent cross-test interfer…
paddybyers Mar 30, 2026
2ad36e9
Rewrite heartbeat test specs and extend mock WebSocket helper
paddybyers Mar 30, 2026
4924343
Fix RTN15a (immediate reconnection) test approach and update skill
paddybyers Mar 30, 2026
5299cbb
Fix minor issues in channel attach and state events test specs
paddybyers Mar 30, 2026
a5438a2
Add test specs for connection state transitions and channel properties
paddybyers Mar 30, 2026
7d8cd91
Add test specs for realtime channel subscribe (RTL7/RTL8)
paddybyers Mar 30, 2026
81fec16
Add test specs for realtime channel publish (RTL6)
paddybyers Mar 30, 2026
0c6b330
Add realtime entries for stats() and time() referencing existing REST…
paddybyers Mar 30, 2026
1b7778c
Add test specs for Realtime.request() (RTN18)
paddybyers Mar 30, 2026
6622edc
Extend realtime publish tests: queued messages and state transitions
paddybyers Mar 30, 2026
0f010d4
Add test specs for RSN1-4 (connection recovery) and RTL10 (realtime h…
paddybyers Mar 30, 2026
4482d5f
Add test specs for client logging (LOG1-LOG3)
paddybyers Mar 30, 2026
156301d
Add remaining auth test specs
paddybyers Mar 30, 2026
31e3b3f
Add test specs for realtime presence (RTP)
paddybyers Mar 30, 2026
bce930b
Fix path component encoding in test specs to use encode_uri_component()
paddybyers Mar 30, 2026
15e211e
Update presence test specs and add integration tests
paddybyers Mar 30, 2026
a0df069
Update write-test-spec skill to emphasise keeping specs in sync
paddybyers Mar 30, 2026
3de2c04
Add test specs for batch presence (RSP4)
paddybyers Mar 30, 2026
dc783aa
Add test specs for token revocation (RSA10)
paddybyers Mar 30, 2026
63e36b4
Add test specs for RTL12 (channel UPDATE event handling)
paddybyers Mar 30, 2026
fed299b
Add test specs for VCDIFF delta message encoding (RTL18/RTL19)
paddybyers Mar 30, 2026
9244325
Add test specs for channel attributes, whenState, timeouts, and auto-…
paddybyers Mar 30, 2026
ec858b3
Add test specs for mutable messages (RTL22/RTL23)
paddybyers Mar 30, 2026
66b1b72
Add test specs for push admin (RSH1/RSH7)
paddybyers Mar 30, 2026
6420ec4
Fix presence test specs: server echoes, wildcard clientId, RTL13b, RT…
paddybyers Mar 30, 2026
14b7d80
Fix integration test specs based on sandbox behavior
paddybyers Mar 30, 2026
9eb85c8
Add CLOSE_CLIENT(client) to all UTS specs that create Realtime clients
paddybyers Apr 30, 2026
1803464
UTS spec corrections from ably-js audit
paddybyers May 2, 2026
564bdf2
RSC10b: Clarify that non-token 401 errors MUST NOT trigger renewal
paddybyers Mar 30, 2026
c20db65
Add realtime integration test specs and fix REST integration specs
paddybyers May 2, 2026
506b77f
Add test proxy framework for integration testing
paddybyers Mar 30, 2026
824a71c
Fix RSL2b3 history time range test to use server timestamps
paddybyers May 3, 2026
9031eb7
Fix mislabeled spec points in proxy integration test specs
paddybyers May 3, 2026
7f85583
Add tier 1+2 proxy integration test specs and proxy writing guidance
paddybyers May 3, 2026
f4751ed
Reorganise UTS docs: move guides to docs/, rewrite README
paddybyers May 3, 2026
c8f5b47
Add UTS specs for missing test coverage
paddybyers May 3, 2026
fcfb8a5
Add RSC15f test: expired fallback not resurrected by late in-flight s…
paddybyers May 3, 2026
22e5304
Add RTN15a proxy test variant for TCP close without WebSocket close f…
paddybyers May 4, 2026
f4cebc8
Remove proxy source code; now lives at ably/uts-proxy
paddybyers May 4, 2026
4b2dce7
UTS: updated completion status
paddybyers May 4, 2026
021fdb3
Fix RSA4c3 test: auth failure while CONNECTED should not set errorReason
paddybyers May 5, 2026
ad58a63
Fix RTN24 tests: connectionId is not inside connectionDetails
paddybyers May 5, 2026
007894c
Fix TM3/TM4 UTS tests: correct spec point interpretations
paddybyers May 5, 2026
1f9efc6
Fix batch operation response format in UTS specs (RSC24, RSA17c)
paddybyers May 5, 2026
7fac103
Add UTS spec for REST fallback proxy integration tests (RSC15l2, RSC1…
paddybyers May 5, 2026
12b06c9
Use endpoint instead of deprecated restHost in proxy fallback spec
paddybyers May 6, 2026
2286d2a
Add comprehensive REST proxy integration tests for HTTP error handling
paddybyers May 6, 2026
4b6317f
Add unique Test IDs to all 1120 tests across 126 UTS spec files
paddybyers May 6, 2026
48b569f
Update RSL1k4 UTS spec: use http_replace_response proxy action
paddybyers May 7, 2026
df587d7
Fix endpoint values in UTS test specs
paddybyers May 7, 2026
b3c0f0f
Update sandbox provisioning URLs to nonprod hostname
paddybyers May 7, 2026
5ac5cb4
Add protocol variant convention to UTS integration specs (G1)
paddybyers May 8, 2026
f355624
Add RSL6a3 UTS spec for msgpack interoperability fixtures
paddybyers May 8, 2026
1f885b6
Add UTS test specs for LiveObjects path-based API (~330 tests)
paddybyers May 13, 2026
f8f1b71
Delegate proxy port assignment to uts-proxy in all test specs
paddybyers May 14, 2026
13be81c
Add REST Objects API specification proposal (RSO*)
paddybyers May 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "submodules/ably-common"]
path = submodules/ably-common
url = https://github.com/ably/ably-common.git
461 changes: 461 additions & 0 deletions specifications/REST-OBJECTS-PROPOSAL.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions submodules/ably-common
Submodule ably-common added at bf2908
104 changes: 104 additions & 0 deletions uts/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# Universal Test Specifications (UTS)

Portable test specifications for Ably client library implementations. Each spec defines what to test in language-neutral pseudocode, which is then translated into runnable tests for each SDK.

## Directory Structure

```
uts/
├── rest/
│ ├── unit/ # REST unit tests (mocked HTTP)
│ │ ├── helpers/
│ │ │ └── mock_http.md # Mock HTTP infrastructure spec
│ │ ├── auth/ # RSA — authentication
│ │ ├── channel/ # RSL — channel operations
│ │ ├── encoding/ # RSL4/RSL6 — message encoding
│ │ ├── presence/ # RSP — REST presence
│ │ ├── push/ # RSH — push admin
│ │ └── types/ # T* — type definitions
│ └── integration/ # REST integration tests (Ably sandbox)
├── realtime/
│ ├── unit/ # Realtime unit tests (mocked WebSocket)
│ │ ├── helpers/
│ │ │ ├── mock_websocket.md # Mock WebSocket infrastructure spec
│ │ │ └── mock_vcdiff.md # Mock VCDiff decoder spec
│ │ ├── auth/ # RSA/RTC8 — realtime auth
│ │ ├── channels/ # RTL/RTS — channels and messages
│ │ ├── client/ # RTC — realtime client
│ │ ├── connection/ # RTN — connection management
│ │ └── presence/ # RTP — realtime presence
│ └── integration/ # Realtime integration tests
│ ├── helpers/
│ │ └── proxy.md # Proxy infrastructure spec
│ ├── proxy/ # Proxy-based fault injection tests
│ └── *.md # Direct sandbox tests
├── docs/ # Guides and reference
│ ├── writing-test-specs.md # How to write UTS specs
│ ├── writing-derived-tests.md # How to translate specs into SDK tests
│ ├── integration-testing.md # Integration testing policy
│ └── completion-status.md # Spec coverage matrix
└── README.md # This file
```

## Spec File Counts

| Category | Count | Description |
|----------|-------|-------------|
| REST unit | 40 | Mocked HTTP client tests |
| REST integration | 11 | Ably sandbox tests |
| Realtime unit | 54 | Mocked WebSocket tests |
| Realtime integration (direct) | 13 | Direct sandbox tests |
| Realtime integration (proxy) | 7 | Fault injection via Go proxy |
| Helper specs | 4 | Mock infrastructure definitions |
| **Total** | **129** | |

## Three Test Tiers

**Unit tests** use mocked transports (MockHttpClient, MockWebSocket) to verify client-side logic: state machines, request formation, response parsing, timer behaviour, error handling. They are fast and deterministic.

**Integration tests** (direct) run against the Ably sandbox to verify that the SDK interoperates correctly with the real service. No fault injection — these test happy-path behaviour and real error responses.

**Integration tests** (proxy) run against the Ably sandbox through a programmable Go proxy that can inject faults: connection drops, suppressed frames, replaced responses, HTTP error injection. These test behaviour that can't be verified without controlling the network path.

## Pseudocode Conventions

Test specs use a consistent pseudocode syntax:

```pseudo
# Setup
mock_http = MockHttpClient(
onConnectionAttempt: (conn) => conn.respond_with_success(),
onRequest: (req) => req.respond_with(200, {"result": "ok"})
)
install_mock(mock_http)
client = Rest(options: ClientOptions(key: "appId.keyId:keySecret"))

# Test
result = AWAIT client.operation()

# Assertions
ASSERT result.field == "ok"
ASSERT request.url.path == "/channels/" + encode_uri_component(name) + "/messages"

# Error testing
AWAIT client.badOperation() FAILS WITH error
ASSERT error.code == 40160

# State transitions
AWAIT_STATE client.connection.state == ConnectionState.connected
```

See [docs/writing-test-specs.md](docs/writing-test-specs.md) for the full pseudocode reference, mock patterns, and conventions.

## Guides

- **[Writing Test Specs](docs/writing-test-specs.md)** — How to author UTS specs: mock patterns, pseudocode conventions, proxy test structure, common mistakes
- **[Writing Derived Tests](docs/writing-derived-tests.md)** — How to translate UTS specs into SDK-specific tests, diagnose failures, and record deviations
- **[Integration Testing Policy](docs/integration-testing.md)** — When to write integration vs unit tests, proxy test design principles, test structure conventions
- **[Completion Status](docs/completion-status.md)** — Coverage matrix tracking which spec items have UTS test specs

## Go Test Proxy

The programmable proxy for integration testing lives in a separate repository: [ably/uts-proxy](https://github.com/ably/uts-proxy). It sits between the SDK and the Ably sandbox, transparently forwarding traffic while allowing rule-based fault injection.

See `realtime/integration/helpers/proxy.md` for the proxy infrastructure specification used by test specs in this repository.
Loading
Loading