|
1 | 1 | # MCP Java SDK Conformance Test Validation Results |
2 | 2 |
|
| 3 | +Last validated: **2026-08-17** against conformance suite |
| 4 | +**`@modelcontextprotocol/conformance@0.2.0-alpha.11`** (SDK at `main`, 2.0.1-SNAPSHOT), targetting |
| 5 | +version 2025-11-25 (`--spec-version 2025-11-25`). |
| 6 | + |
3 | 7 | ## Summary |
4 | 8 |
|
5 | | -**Server Tests (active suite):** 44/44 passed (31 scenarios, 100%) |
6 | | -**Server Tests (spec 2025-11-25):** 4/4 passed — SEP-1613 `json-schema-2020-12` scenario ✨ |
7 | | -**Client Tests:** 3/4 scenarios passed (9/10 checks passed) |
8 | | -**Auth Tests:** 15/15 scenarios fully passing (195 passed, 0 failed, 0 warnings, 100% scenarios, 100% checks) |
| 9 | +**Server Tests (active suite):** 73/73 checks passed (31 scenarios, 100%) |
| 10 | +**Server Tests (SEP-1613 `json-schema-2020-12`):** 5/5 checks passed (SEP-2106 checks skipped — post-2025-11-25 spec additions) |
| 11 | +**Client Tests:** 3/4 scenarios passed; `sse-retry` fails (tracked in `conformance-baseline.yml`) |
| 12 | +**Auth Tests:** 14/14 scenarios passing (193 checks, 0 failed, 0 warnings) |
| 13 | + |
| 14 | +Baseline check passed on every run: all failures are expected per |
| 15 | +[`conformance-baseline.yml`](conformance-baseline.yml). |
9 | 16 |
|
10 | 17 | ## Server Test Results |
11 | 18 |
|
12 | | -### Active Suite — Passing (31/31 scenarios, 44/44 checks) |
| 19 | +### Active Suite — Passing (31/31 scenarios, 73/73 checks) |
13 | 20 |
|
14 | | -- **Lifecycle & Utilities (4/4):** initialize, ping, logging-set-level, completion-complete |
15 | | -- **Tools (13/13):** All scenarios including progress notifications, sampling, elicitation ✨ |
16 | | -- **Elicitation (10/10):** SEP-1034 defaults (5 checks), SEP-1330 enums (5 checks) |
17 | | -- **Resources (7/7):** list, read-text, read-binary, templates-read, subscribe, unsubscribe, SEP-2164 resource-not-found |
18 | | -- **Prompts (5/5):** list, simple, with-args, embedded-resource, with-image |
19 | | -- **SSE Transport (2/2):** Multiple streams |
20 | | -- **Security (2/2):** Localhost validation passes, DNS rebinding protection |
| 21 | +- **Lifecycle & Utilities:** initialize, ping, logging-set-level, completion-complete |
| 22 | +- **Tools (13/13):** all scenarios including progress notifications, sampling, elicitation |
| 23 | +- **Elicitation:** SEP-1034 defaults (6 checks), SEP-1330 enums (6 checks) |
| 24 | +- **Resources:** list, read-text, read-binary, templates-read, subscribe, unsubscribe |
| 25 | +- **Prompts:** list, simple, with-args, embedded-resource, with-image |
| 26 | +- **SSE Transport:** multiple streams |
| 27 | +- **Security:** DNS rebinding protection |
21 | 28 |
|
22 | | -### Spec 2025-11-25 Scenarios — Passing (1/1 scenario, 4/4 checks) |
| 29 | +### SEP-1613 — JSON Schema 2020-12 (5/5 checks) |
23 | 30 |
|
24 | | -- **JSON Schema 2020-12 (SEP-1613) (4/4):** ✨ |
25 | | - - `json_schema_2020_12_tool` found |
26 | | - - `inputSchema.$schema` field preserved |
27 | | - - `inputSchema.$defs` field preserved |
28 | | - - `inputSchema.additionalProperties` field preserved |
| 31 | +- `json_schema_2020_12_tool` found; `$schema`, `$defs`, and `additionalProperties` |
| 32 | + fields preserved; every JSON-RPC message valid per the spec JSON schema for the |
| 33 | + negotiated spec version (`wire-schema-valid`) |
| 34 | +- SEP-2106 checks (composition/conditional/anchor keywords) reported SKIPPED: |
| 35 | + they postdate the 2025-11-25 spec release and are excluded from scoring |
29 | 36 |
|
30 | 37 | ## Client Test Results |
31 | 38 |
|
32 | | -### Passing (3/4 scenarios, 9/10 checks) |
33 | | - |
34 | | -- **initialize (1/1):** Protocol negotiation, clientInfo, capabilities |
35 | | -- **tools_call (1/1):** Tool discovery and invocation |
36 | | -- **elicitation-sep1034-client-defaults (5/5):** Default values for string, integer, number, enum, boolean |
| 39 | +### Passing (3/4 scenarios) |
37 | 40 |
|
38 | | -### Partially Passing (1/4 scenarios, 1/2 checks) |
| 41 | +- **initialize (1/1):** protocol negotiation, clientInfo, capabilities |
| 42 | +- **tools_call (2/2):** tool discovery and invocation |
| 43 | +- **elicitation-sep1034-client-defaults (5/5):** default values for string, integer, number, enum, boolean |
39 | 44 |
|
40 | | -- **sse-retry (1/2 + 1 warning):** |
41 | | - - ✅ Reconnects after stream closure |
42 | | - - ❌ Does not respect retry timing |
43 | | - - ⚠️ Does not send Last-Event-ID header (SHOULD requirement) |
| 45 | +### Failing — in baseline (1/4 scenarios) |
44 | 46 |
|
45 | | -**Issue:** Client treats `retry:` SSE field as invalid instead of parsing it for reconnection timing. |
| 47 | +- **sse-retry:** client does not parse/respect the `retry:` SSE field timing and |
| 48 | + does not send the `Last-Event-ID` header (SHOULD requirement). Expected failure, |
| 49 | + listed in `conformance-baseline.yml`. |
46 | 50 |
|
47 | 51 | ## Auth Test Results (Spring HTTP Client) |
48 | 52 |
|
49 | | -**Status: 195 passed, 0 failed, 0 warnings across 15 scenarios** |
| 53 | +**Status: 193 checks passed, 0 failed, 0 warnings across 14 scenarios** |
50 | 54 |
|
51 | | -Uses the `client-spring-http-client` module with Spring Security OAuth2 and the [mcp-client-security](https://github.com/springaicommunity/mcp-client-security) library. |
| 55 | +Uses the `client-spring-http-client` module with Spring Security OAuth2 and the |
| 56 | +[mcp-client-security](https://github.com/springaicommunity/mcp-client-security) library. |
52 | 57 |
|
53 | | -### Fully Passing (15/15 scenarios) |
| 58 | +Fully passing: metadata-default, metadata-var1/2/3, basic-cimd, |
| 59 | +scope-from-www-authenticate, scope-from-scopes-supported, scope-omitted-when-undefined, |
| 60 | +scope-step-up, scope-retry-limit, token-endpoint-auth-basic/post/none, pre-registration. |
54 | 61 |
|
55 | | -- **auth/metadata-default (13/13):** Default metadata discovery |
56 | | -- **auth/metadata-var1 (13/13):** Metadata discovery variant 1 |
57 | | -- **auth/metadata-var2 (13/13):** Metadata discovery variant 2 |
58 | | -- **auth/metadata-var3 (13/13):** Metadata discovery variant 3 |
59 | | -- **auth/basic-cimd (12/12):** Basic Client-Initiated Metadata Discovery |
60 | | -- **auth/scope-from-www-authenticate (14/14):** Scope extraction from WWW-Authenticate header |
61 | | -- **auth/scope-from-scopes-supported (14/14):** Scope extraction from scopes_supported |
62 | | -- **auth/scope-omitted-when-undefined (14/14):** Scope omitted when not defined |
63 | | -- **auth/scope-step-up (16/16):** Scope step-up challenge |
64 | | -- **auth/scope-retry-limit (11/11):** Scope retry limit handling |
65 | | -- **auth/token-endpoint-auth-basic (18/18):** Token endpoint with HTTP Basic auth |
66 | | -- **auth/token-endpoint-auth-post (18/18):** Token endpoint with POST body auth |
67 | | -- **auth/token-endpoint-auth-none (18/18):** Token endpoint with no client auth |
68 | | -- **auth/resource-mismatch (2/2):** Resource mismatch handling |
69 | | -- **auth/pre-registration (6/6):** Pre-registered client credentials flow |
| 62 | +Note: `auth/resource-mismatch` (present in earlier suite versions) is no longer part |
| 63 | +of the 0.2.0-alpha auth suite. |
70 | 64 |
|
71 | 65 | ## Known Limitations |
72 | 66 |
|
73 | | -1. **Client SSE Retry:** Client doesn't parse or respect the `retry:` field, reconnects immediately, and doesn't send Last-Event-ID header |
| 67 | +1. **Client SSE Retry:** client doesn't parse or respect the `retry:` field, |
| 68 | + reconnects immediately, and doesn't send the `Last-Event-ID` header |
74 | 69 |
|
75 | 70 | ## Running Tests |
76 | 71 |
|
77 | 72 | ### Server (active suite) |
78 | 73 | ```bash |
79 | | -# Start server |
80 | | -./mvnw compile -pl conformance-tests/server-servlet -am exec:java |
81 | | - |
82 | | -# Run tests (in another terminal) |
83 | | -npx @modelcontextprotocol/conformance server --url http://localhost:8080/mcp --suite active |
| 74 | +# Build and start server |
| 75 | +./mvnw clean install -DskipTests |
| 76 | +mvn exec:java -pl conformance-tests/server-servlet \ |
| 77 | + -Dexec.mainClass="io.modelcontextprotocol.conformance.server.ConformanceServlet" |
| 78 | + |
| 79 | +# Run tests (in another terminal, from the repo root) |
| 80 | +npx @modelcontextprotocol/conformance@0.2.0-alpha.11 server \ |
| 81 | + --url http://localhost:8080/mcp --suite active \ |
| 82 | + --expected-failures ./conformance-tests/conformance-baseline.yml |
84 | 83 | ``` |
85 | 84 |
|
86 | | -### Server (spec 2025-11-25 scenarios — SEP-1613) |
| 85 | +### Server (SEP-1613 scenario) |
87 | 86 | ```bash |
88 | | -# Start server (if not already running) |
89 | | -./mvnw compile -pl conformance-tests/server-servlet -am exec:java |
90 | | - |
91 | | -# Run json-schema-2020-12 scenario |
92 | | -cd ../conformance && node --import tsx/esm src/index.ts server \ |
93 | | - --url http://localhost:8080/mcp \ |
94 | | - --scenario json-schema-2020-12 |
| 87 | +npx @modelcontextprotocol/conformance@0.2.0-alpha.11 server \ |
| 88 | + --url http://localhost:8080/mcp --scenario json-schema-2020-12 |
95 | 89 | ``` |
96 | 90 |
|
97 | 91 | ### Client |
98 | 92 | ```bash |
99 | | -# Build |
100 | | -cd conformance-tests/client-jdk-http-client |
101 | | -../../mvnw clean package -DskipTests |
102 | | - |
103 | | -# Run all scenarios |
104 | 93 | for scenario in initialize tools_call elicitation-sep1034-client-defaults sse-retry; do |
105 | | - npx @modelcontextprotocol/conformance client \ |
106 | | - --command "java -jar target/client-jdk-http-client-1.1.0-SNAPSHOT.jar" \ |
107 | | - --scenario $scenario |
| 94 | + npx @modelcontextprotocol/conformance@0.2.0-alpha.11 client \ |
| 95 | + --command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-*.jar" \ |
| 96 | + --scenario $scenario \ |
| 97 | + --expected-failures ./conformance-tests/conformance-baseline.yml |
108 | 98 | done |
109 | 99 | ``` |
110 | 100 |
|
111 | 101 | ### Auth (Spring HTTP Client) |
112 | | - |
113 | | -Ensure you run with the conformance testing suite `0.1.15` or higher. |
114 | | - |
115 | 102 | ```bash |
116 | | -# Build |
117 | | -cd conformance-tests/client-spring-http-client |
118 | | -../../mvnw clean package -DskipTests |
119 | | - |
120 | | -# Run auth suite |
121 | | -npx @modelcontextprotocol/conformance@0.1.15 client \ |
| 103 | +npx @modelcontextprotocol/conformance@0.2.0-alpha.11 client \ |
122 | 104 | --spec-version 2025-11-25 \ |
123 | | - --command "java -jar target/client-spring-http-client-1.1.0-SNAPSHOT.jar" \ |
124 | | - --suite auth |
| 105 | + --command "java -jar conformance-tests/client-spring-http-client/target/client-spring-http-client-*.jar" \ |
| 106 | + --suite auth \ |
| 107 | + --expected-failures ./conformance-tests/conformance-baseline.yml |
125 | 108 | ``` |
126 | 109 |
|
127 | 110 | ## Recommendations |
|
0 commit comments