Add E2E scenario tests/examples for all SDK languages#512
Conversation
Add csharp/ subdirectories with Program.cs and csproj for: - transport/stdio, transport/tcp - bundling/fully-bundled - tools/no-tools, tools/custom-agents - sessions/streaming - callbacks/permissions - prompts/system-message All samples reference the local .NET SDK via ProjectReference. Each verify.sh updated with dotnet build/run steps. Added C# build artifacts to .gitignore. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
All 30 go.mod files had incorrect relative paths to the SDK Go module. Scenarios live at test/scenarios/<cat>/<scenario>/go/ (5 levels deep), so the replace directive needs ../../../../../go instead of ../../../../go. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
copilot-core is not a product name. Updated 63 files to use the correct terminology: - Prose/comments: "Copilot CLI" - Binary name in code/commands: copilot - COPILOT_CLI_PATH env var: unchanged (already correct) - Dockerfile ENTRYPOINT/COPY: copilot - docker-compose service name: copilot-cli Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Runs build verification for all test/scenarios across 4 languages (TypeScript, Python, Go, C#) as parallel jobs. Triggered on PRs that touch scenarios or SDK source, and on push to main. Build-only — no E2E execution (no API keys or Copilot CLI needed). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Every scenario now has TypeScript, Python, Go, and C# implementations. New additions: - C# for 26 scenarios (auth, bundling, callbacks, modes, prompts, sessions, tools, transport) - Python + Go + C# for 3 BYOK scenarios (anthropic, azure, ollama) - Python + Go + C# stubs for multi-user scenarios (SKIP pattern) All 136 scenario builds verified: 34 TS, 34 PY, 34 GO, 34 CS. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add npm, Go module, and NuGet caching to scenario-builds.yml - Add just scenario-build, scenario-verify, scenario-build-lang targets Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
patniko
changed the title
There was a problem hiding this comment.
Pull request overview
Adds a large suite of end-to-end “scenario” samples under test/scenarios/ to verify SDK behavior consistently across TypeScript, Python, Go, and C#.
Changes:
- Introduces many new per-scenario, per-language runnable samples (client/session/tools/callback flows).
- Adds per-scenario docs plus supporting build metadata (e.g.,
package.json,go.mod,*.csproj,tsconfig.json). - Adds scenario-level ignore rules for build/run artifacts.
Reviewed changes
Copilot reviewed 274 out of 385 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| test/scenarios/tools/mcp-servers/go/main.go | Go sample demonstrating MCP server configuration in session config |
| test/scenarios/tools/mcp-servers/go/go.mod | Go module setup wiring scenario to local Go SDK via replace |
| test/scenarios/tools/mcp-servers/csharp/csharp.csproj | C# project for MCP servers scenario referencing .NET SDK project |
| test/scenarios/tools/mcp-servers/csharp/Program.cs | C# sample demonstrating MCP servers session config |
| test/scenarios/tools/mcp-servers/README.md | Scenario documentation for MCP servers configuration |
| test/scenarios/tools/custom-agents/typescript/src/index.ts | TypeScript sample demonstrating customAgents session config |
| test/scenarios/tools/custom-agents/typescript/package.json | TypeScript build/run config for custom-agents sample |
| test/scenarios/tools/custom-agents/python/requirements.txt | Python sample dependency wiring to local Python SDK |
| test/scenarios/tools/custom-agents/python/main.py | Python sample demonstrating custom_agents session config |
| test/scenarios/tools/custom-agents/go/main.go | Go sample demonstrating CustomAgents session config |
| test/scenarios/tools/custom-agents/go/go.mod | Go module setup for custom-agents scenario |
| test/scenarios/tools/custom-agents/csharp/csharp.csproj | C# project for custom-agents scenario |
| test/scenarios/tools/custom-agents/csharp/Program.cs | C# sample demonstrating CustomAgents session config |
| test/scenarios/tools/custom-agents/README.md | Scenario documentation for custom agents |
| test/scenarios/sessions/streaming/typescript/src/index.ts | TypeScript sample enabling streaming and counting stream events |
| test/scenarios/sessions/streaming/typescript/package.json | TypeScript build/run config for streaming scenario |
| test/scenarios/sessions/streaming/python/requirements.txt | Python dependency wiring for streaming scenario |
| test/scenarios/sessions/streaming/python/main.py | Python sample enabling streaming and counting stream events |
| test/scenarios/sessions/streaming/go/main.go | Go sample enabling streaming and counting stream events |
| test/scenarios/sessions/streaming/go/go.mod | Go module setup for streaming scenario |
| test/scenarios/sessions/streaming/csharp/csharp.csproj | C# project for streaming scenario |
| test/scenarios/sessions/streaming/csharp/Program.cs | C# sample enabling streaming and counting delta events |
| test/scenarios/sessions/streaming/README.md | Scenario documentation for streaming behavior |
| test/scenarios/sessions/session-resume/typescript/src/index.ts | TypeScript sample for session persistence/resume flow |
| test/scenarios/sessions/session-resume/typescript/package.json | TypeScript build/run config for session-resume scenario |
| test/scenarios/sessions/session-resume/python/requirements.txt | Python dependency wiring for session-resume scenario |
| test/scenarios/sessions/session-resume/python/main.py | Python sample for session persistence/resume flow |
| test/scenarios/sessions/session-resume/go/main.go | Go sample for session persistence/resume flow |
| test/scenarios/sessions/session-resume/go/go.mod | Go module setup for session-resume scenario |
| test/scenarios/sessions/session-resume/csharp/csharp.csproj | C# project for session-resume scenario |
| test/scenarios/sessions/session-resume/csharp/Program.cs | C# sample for session persistence/resume flow |
| test/scenarios/sessions/session-resume/README.md | Scenario documentation for session resume |
| test/scenarios/sessions/multi-user-short-lived/typescript/src/index.ts | TypeScript placeholder marking scenario as skipped |
| test/scenarios/sessions/multi-user-short-lived/typescript/package.json | TypeScript setup for multi-user-short-lived scenario |
| test/scenarios/sessions/multi-user-short-lived/python/requirements.txt | Python dependency wiring for multi-user-short-lived scenario |
| test/scenarios/sessions/multi-user-short-lived/python/main.py | Python placeholder marking scenario as skipped |
| test/scenarios/sessions/multi-user-short-lived/go/main.go | Go placeholder marking scenario as skipped |
| test/scenarios/sessions/multi-user-short-lived/go/go.mod | Go module metadata for skipped multi-user-short-lived |
| test/scenarios/sessions/multi-user-short-lived/csharp/csharp.csproj | C# project for multi-user-short-lived scenario |
| test/scenarios/sessions/multi-user-short-lived/csharp/Program.cs | C# placeholder marking scenario as skipped |
| test/scenarios/sessions/multi-user-short-lived/README.md | Scenario documentation for multi-user short-lived pattern |
| test/scenarios/sessions/multi-user-long-lived/typescript/src/index.ts | TypeScript placeholder marking scenario as skipped |
| test/scenarios/sessions/multi-user-long-lived/typescript/package.json | TypeScript setup for multi-user-long-lived scenario |
| test/scenarios/sessions/multi-user-long-lived/python/requirements.txt | Python dependency wiring for multi-user-long-lived scenario |
| test/scenarios/sessions/multi-user-long-lived/python/main.py | Python placeholder marking scenario as skipped |
| test/scenarios/sessions/multi-user-long-lived/go/main.go | Go placeholder marking scenario as skipped |
| test/scenarios/sessions/multi-user-long-lived/go/go.mod | Go module metadata for skipped multi-user-long-lived |
| test/scenarios/sessions/multi-user-long-lived/csharp/csharp.csproj | C# project for multi-user-long-lived scenario |
| test/scenarios/sessions/multi-user-long-lived/csharp/Program.cs | C# placeholder marking scenario as skipped |
| test/scenarios/sessions/multi-user-long-lived/README.md | Scenario documentation for multi-user long-lived pattern |
| test/scenarios/sessions/infinite-sessions/typescript/src/index.ts | TypeScript placeholder marking scenario as skipped |
| test/scenarios/sessions/infinite-sessions/typescript/package.json | TypeScript setup for infinite-sessions scenario |
| test/scenarios/sessions/infinite-sessions/python/requirements.txt | Python dependency wiring for infinite-sessions scenario |
| test/scenarios/sessions/infinite-sessions/python/main.py | Python sample enabling infinite sessions config |
| test/scenarios/sessions/infinite-sessions/go/main.go | Go sample enabling infinite sessions config |
| test/scenarios/sessions/infinite-sessions/go/go.mod | Go module setup for infinite-sessions scenario |
| test/scenarios/sessions/infinite-sessions/csharp/csharp.csproj | C# project for infinite-sessions scenario |
| test/scenarios/sessions/infinite-sessions/csharp/Program.cs | C# sample enabling infinite sessions config |
| test/scenarios/sessions/infinite-sessions/README.md | Scenario documentation for infinite sessions |
| test/scenarios/sessions/concurrent-sessions/typescript/src/index.ts | TypeScript sample creating sessions concurrently to validate isolation |
| test/scenarios/sessions/concurrent-sessions/typescript/package.json | TypeScript build/run config for concurrent-sessions scenario |
| test/scenarios/sessions/concurrent-sessions/python/requirements.txt | Python dependency wiring for concurrent-sessions scenario |
| test/scenarios/sessions/concurrent-sessions/python/main.py | Python sample for concurrent sessions isolation |
| test/scenarios/sessions/concurrent-sessions/go/main.go | Go sample for concurrent sessions isolation |
| test/scenarios/sessions/concurrent-sessions/go/go.mod | Go module setup for concurrent-sessions scenario |
| test/scenarios/sessions/concurrent-sessions/csharp/csharp.csproj | C# project for concurrent-sessions scenario |
| test/scenarios/sessions/concurrent-sessions/csharp/Program.cs | C# sample for concurrent sessions isolation |
| test/scenarios/sessions/concurrent-sessions/README.md | Scenario documentation for concurrent sessions |
| test/scenarios/prompts/system-message/typescript/src/index.ts | TypeScript sample demonstrating systemMessage replace |
| test/scenarios/prompts/system-message/typescript/package.json | TypeScript setup for system-message scenario |
| test/scenarios/prompts/system-message/python/requirements.txt | Python dependency wiring for system-message scenario |
| test/scenarios/prompts/system-message/python/main.py | Python sample demonstrating system_message replace |
| test/scenarios/prompts/system-message/go/main.go | Go sample demonstrating SystemMessage replace |
| test/scenarios/prompts/system-message/go/go.mod | Go module setup for system-message scenario |
| test/scenarios/prompts/system-message/csharp/csharp.csproj | C# project for system-message scenario |
| test/scenarios/prompts/system-message/csharp/Program.cs | C# sample demonstrating SystemMessage replace |
| test/scenarios/prompts/system-message/README.md | Scenario documentation for system messages |
| test/scenarios/prompts/reasoning-effort/typescript/src/index.ts | TypeScript sample setting reasoningEffort |
| test/scenarios/prompts/reasoning-effort/typescript/package.json | TypeScript setup for reasoning-effort scenario |
| test/scenarios/prompts/reasoning-effort/python/requirements.txt | Python dependency wiring for reasoning-effort scenario |
| test/scenarios/prompts/reasoning-effort/python/main.py | Python sample setting reasoning_effort |
| test/scenarios/prompts/reasoning-effort/go/main.go | Go sample setting ReasoningEffort |
| test/scenarios/prompts/reasoning-effort/go/go.mod | Go module setup for reasoning-effort scenario |
| test/scenarios/prompts/reasoning-effort/csharp/csharp.csproj | C# project for reasoning-effort scenario |
| test/scenarios/prompts/reasoning-effort/csharp/Program.cs | C# sample setting ReasoningEffort |
| test/scenarios/prompts/reasoning-effort/README.md | Scenario documentation for reasoning effort |
| test/scenarios/prompts/attachments/typescript/src/index.ts | TypeScript sample sending file attachments |
| test/scenarios/prompts/attachments/typescript/package.json | TypeScript setup for attachments scenario |
| test/scenarios/prompts/attachments/sample-data.txt | Sample data file used as attachment |
| test/scenarios/prompts/attachments/python/requirements.txt | Python dependency wiring for attachments scenario |
| test/scenarios/prompts/attachments/python/main.py | Python sample sending file attachments |
| test/scenarios/prompts/attachments/go/main.go | Go sample sending file attachments |
| test/scenarios/prompts/attachments/go/go.mod | Go module setup for attachments scenario |
| test/scenarios/prompts/attachments/csharp/csharp.csproj | C# project for attachments scenario |
| test/scenarios/prompts/attachments/csharp/Program.cs | C# sample sending file attachments |
| test/scenarios/prompts/attachments/README.md | Scenario documentation for file attachments |
| test/scenarios/modes/minimal-preset/typescript/src/index.ts | TypeScript placeholder marking scenario as skipped |
| test/scenarios/modes/minimal-preset/typescript/package.json | TypeScript setup for minimal-preset scenario |
| test/scenarios/modes/minimal-preset/python/requirements.txt | Python dependency wiring for minimal-preset scenario |
| test/scenarios/modes/minimal-preset/python/main.py | Python placeholder marking scenario as skipped |
| test/scenarios/modes/minimal-preset/go/main.go | Go placeholder marking scenario as skipped |
| test/scenarios/modes/minimal-preset/go/go.mod | Go module setup for minimal-preset scenario |
| test/scenarios/modes/minimal-preset/csharp/csharp.csproj | C# project for minimal-preset scenario |
| test/scenarios/modes/minimal-preset/csharp/Program.cs | C# placeholder marking scenario as skipped |
| test/scenarios/modes/filesystem-preset/typescript/src/index.ts | TypeScript placeholder marking scenario as skipped |
| test/scenarios/modes/filesystem-preset/typescript/package.json | TypeScript setup for filesystem-preset scenario |
| test/scenarios/modes/filesystem-preset/python/requirements.txt | Python dependency wiring for filesystem-preset scenario |
| test/scenarios/modes/filesystem-preset/python/main.py | Python placeholder marking scenario as skipped |
| test/scenarios/modes/filesystem-preset/go/main.go | Go placeholder marking scenario as skipped |
| test/scenarios/modes/filesystem-preset/go/go.mod | Go module setup for filesystem-preset scenario |
| test/scenarios/modes/filesystem-preset/csharp/csharp.csproj | C# project for filesystem-preset scenario |
| test/scenarios/modes/filesystem-preset/csharp/Program.cs | C# placeholder marking scenario as skipped |
| test/scenarios/modes/cli-preset/typescript/src/index.ts | TypeScript placeholder marking scenario as skipped |
| test/scenarios/modes/cli-preset/typescript/package.json | TypeScript setup for cli-preset scenario |
| test/scenarios/modes/cli-preset/python/requirements.txt | Python dependency wiring for cli-preset scenario |
| test/scenarios/modes/cli-preset/python/main.py | Python placeholder marking scenario as skipped |
| test/scenarios/modes/cli-preset/go/main.go | Go placeholder marking scenario as skipped |
| test/scenarios/modes/cli-preset/go/go.mod | Go module setup for cli-preset scenario |
| test/scenarios/modes/cli-preset/csharp/csharp.csproj | C# project for cli-preset scenario |
| test/scenarios/modes/cli-preset/csharp/Program.cs | C# placeholder marking scenario as skipped |
| test/scenarios/callbacks/user-input/typescript/src/index.ts | TypeScript sample exercising ask_user callback flow |
| test/scenarios/callbacks/user-input/typescript/package.json | TypeScript setup for user-input scenario |
| test/scenarios/callbacks/user-input/python/requirements.txt | Python dependency wiring for user-input scenario |
| test/scenarios/callbacks/user-input/python/main.py | Python sample exercising ask_user callback flow |
| test/scenarios/callbacks/user-input/go/main.go | Go sample exercising ask_user callback flow |
| test/scenarios/callbacks/user-input/go/go.mod | Go module setup for user-input scenario |
| test/scenarios/callbacks/user-input/csharp/csharp.csproj | C# project for user-input scenario |
| test/scenarios/callbacks/user-input/csharp/Program.cs | C# sample exercising ask_user callback flow |
| test/scenarios/callbacks/user-input/README.md | Scenario documentation for user input callback |
| test/scenarios/callbacks/permissions/typescript/src/index.ts | TypeScript sample exercising onPermissionRequest hook |
| test/scenarios/callbacks/permissions/typescript/package.json | TypeScript setup for permissions scenario |
| test/scenarios/callbacks/permissions/python/requirements.txt | Python dependency wiring for permissions scenario |
| test/scenarios/callbacks/permissions/python/main.py | Python sample exercising on_permission_request hook |
| test/scenarios/callbacks/permissions/go/main.go | Go sample exercising OnPermissionRequest hook |
| test/scenarios/callbacks/permissions/go/go.mod | Go module setup for permissions scenario |
| test/scenarios/callbacks/permissions/csharp/csharp.csproj | C# project for permissions scenario |
| test/scenarios/callbacks/permissions/csharp/Program.cs | C# sample exercising permission request flow |
| test/scenarios/callbacks/permissions/README.md | Scenario documentation for permission request flow |
| test/scenarios/callbacks/hooks/typescript/src/index.ts | TypeScript sample exercising lifecycle hooks |
| test/scenarios/callbacks/hooks/typescript/package.json | TypeScript setup for hooks scenario |
| test/scenarios/callbacks/hooks/python/requirements.txt | Python dependency wiring for hooks scenario |
| test/scenarios/callbacks/hooks/python/main.py | Python sample exercising lifecycle hooks |
| test/scenarios/callbacks/hooks/go/main.go | Go sample exercising lifecycle hooks |
| test/scenarios/callbacks/hooks/go/go.mod | Go module setup for hooks scenario |
| test/scenarios/callbacks/hooks/csharp/csharp.csproj | C# project for hooks scenario |
| test/scenarios/callbacks/hooks/csharp/Program.cs | C# sample exercising lifecycle hooks |
| test/scenarios/callbacks/hooks/README.md | Scenario documentation for lifecycle hooks |
| test/scenarios/bundling/fully-bundled/typescript/tsconfig.json | TS compiler config for fully-bundled sample |
| test/scenarios/bundling/fully-bundled/typescript/src/index.ts | TypeScript fully-bundled sample code |
| test/scenarios/bundling/fully-bundled/typescript/package.json | TypeScript setup for fully-bundled sample |
| test/scenarios/bundling/fully-bundled/python/requirements.txt | Python dependency wiring for fully-bundled sample |
| test/scenarios/bundling/fully-bundled/python/main.py | Python fully-bundled sample code |
| test/scenarios/bundling/fully-bundled/go/main.go | Go fully-bundled sample code |
| test/scenarios/bundling/fully-bundled/go/go.mod | Go module setup for fully-bundled sample |
| test/scenarios/bundling/fully-bundled/csharp/csharp.csproj | C# project for fully-bundled sample |
| test/scenarios/bundling/fully-bundled/csharp/Program.cs | C# fully-bundled sample code |
| test/scenarios/bundling/fully-bundled/README.md | Documentation for fully-bundled deployment pattern |
| test/scenarios/bundling/container-proxy/typescript/tsconfig.json | TS compiler config for container-proxy sample |
| test/scenarios/bundling/container-proxy/typescript/src/index.ts | TypeScript client sample for container-proxy pattern |
| test/scenarios/bundling/container-proxy/typescript/package.json | TypeScript setup for container-proxy sample |
| test/scenarios/bundling/container-proxy/python/requirements.txt | Python dependency wiring for container-proxy sample |
| test/scenarios/bundling/container-proxy/python/main.py | Python client sample for container-proxy pattern |
| test/scenarios/bundling/container-proxy/go/main.go | Go client sample for container-proxy pattern |
| test/scenarios/bundling/container-proxy/go/go.mod | Go module setup for container-proxy sample |
| test/scenarios/bundling/container-proxy/docker-compose.yml | Docker compose to run Copilot CLI in container |
| test/scenarios/bundling/container-proxy/csharp/csharp.csproj | C# project for container-proxy sample |
| test/scenarios/bundling/container-proxy/csharp/Program.cs | C# client sample for container-proxy pattern |
| test/scenarios/bundling/container-proxy/Dockerfile | Container image definition for running Copilot CLI |
| test/scenarios/bundling/container-proxy/.dockerignore | Docker build context filtering for container-proxy |
| test/scenarios/bundling/app-direct-server/typescript/tsconfig.json | TS compiler config for app-direct-server sample |
| test/scenarios/bundling/app-direct-server/typescript/src/index.ts | TypeScript client connecting to pre-running CLI server |
| test/scenarios/bundling/app-direct-server/typescript/package.json | TypeScript setup for app-direct-server sample |
| test/scenarios/bundling/app-direct-server/python/requirements.txt | Python dependency wiring for app-direct-server sample |
| test/scenarios/bundling/app-direct-server/python/main.py | Python client connecting to pre-running CLI server |
| test/scenarios/bundling/app-direct-server/go/main.go | Go client connecting to pre-running CLI server |
| test/scenarios/bundling/app-direct-server/go/go.mod | Go module setup for app-direct-server sample |
| test/scenarios/bundling/app-direct-server/csharp/csharp.csproj | C# project for app-direct-server sample |
| test/scenarios/bundling/app-direct-server/csharp/Program.cs | C# client connecting to pre-running CLI server |
| test/scenarios/bundling/app-direct-server/README.md | Documentation for app-direct-server deployment pattern |
| test/scenarios/bundling/app-backend-to-server/typescript/src/index.ts | TypeScript Express backend proxying to Copilot CLI server |
| test/scenarios/bundling/app-backend-to-server/typescript/package.json | TypeScript deps/scripts for Express backend sample |
| test/scenarios/bundling/app-backend-to-server/python/requirements.txt | Python deps for Flask backend sample |
| test/scenarios/bundling/app-backend-to-server/python/main.py | Python Flask backend proxying to Copilot CLI server |
| test/scenarios/bundling/app-backend-to-server/go/go.mod | Go module placeholder for app-backend-to-server scenario |
| test/scenarios/bundling/app-backend-to-server/csharp/csharp.csproj | C# project for app-backend-to-server scenario |
| test/scenarios/bundling/app-backend-to-server/csharp/Program.cs | C# sample connecting to Copilot CLI server |
| test/scenarios/auth/token-sources/typescript/src/index.ts | TypeScript sample demonstrating token source resolution |
| test/scenarios/auth/token-sources/typescript/package.json | TypeScript setup for token-sources scenario |
| test/scenarios/auth/token-sources/python/requirements.txt | Python dependency wiring for token-sources scenario |
| test/scenarios/auth/token-sources/python/main.py | Python sample demonstrating token source resolution |
| test/scenarios/auth/token-sources/go/main.go | Go sample demonstrating token source resolution |
| test/scenarios/auth/token-sources/go/go.mod | Go module setup for token-sources scenario |
| test/scenarios/auth/token-sources/csharp/csharp.csproj | C# project for token-sources scenario |
| test/scenarios/auth/token-sources/csharp/Program.cs | C# sample demonstrating token source resolution |
| test/scenarios/auth/token-sources/README.md | Documentation for token source priority chain |
| test/scenarios/auth/gh-app/typescript/package.json | TypeScript setup for GitHub OAuth device flow scenario |
| test/scenarios/auth/gh-app/python/requirements.txt | Python dependency wiring for gh-app scenario |
| test/scenarios/auth/gh-app/python/main.py | Python interactive OAuth device flow sample |
| test/scenarios/auth/gh-app/go/go.mod | Go module setup for gh-app scenario |
| test/scenarios/auth/gh-app/csharp/csharp.csproj | C# project for gh-app scenario |
| test/scenarios/auth/gh-app/csharp/Program.cs | C# interactive OAuth device flow sample |
| test/scenarios/auth/gh-app/README.md | Documentation for GitHub OAuth device flow scenario |
| test/scenarios/auth/byok-openai/typescript/src/index.ts | TypeScript BYOK OpenAI provider sample |
| test/scenarios/auth/byok-openai/typescript/package.json | TypeScript setup for byok-openai scenario |
| test/scenarios/auth/byok-openai/python/requirements.txt | Python dependency wiring for byok-openai scenario |
| test/scenarios/auth/byok-openai/python/main.py | Python BYOK OpenAI provider sample |
| test/scenarios/auth/byok-openai/go/main.go | Go BYOK OpenAI provider sample |
| test/scenarios/auth/byok-openai/go/go.mod | Go module setup for byok-openai scenario |
| test/scenarios/auth/byok-openai/csharp/csharp.csproj | C# project for byok-openai scenario |
| test/scenarios/auth/byok-openai/csharp/Program.cs | C# BYOK OpenAI provider sample |
| test/scenarios/auth/byok-openai/README.md | Documentation for BYOK OpenAI scenario |
| test/scenarios/auth/byok-ollama/verify.sh | Per-scenario verification script for Ollama BYOK |
| test/scenarios/auth/byok-ollama/typescript/src/index.ts | TypeScript BYOK Ollama provider sample |
| test/scenarios/auth/byok-ollama/typescript/package.json | TypeScript setup for byok-ollama scenario |
| test/scenarios/auth/byok-ollama/python/requirements.txt | Python dependency wiring for byok-ollama scenario |
| test/scenarios/auth/byok-ollama/python/main.py | Python BYOK Ollama provider sample |
| test/scenarios/auth/byok-ollama/go/main.go | Go BYOK Ollama provider sample |
| test/scenarios/auth/byok-ollama/go/go.mod | Go module setup for byok-ollama scenario |
| test/scenarios/auth/byok-ollama/csharp/csharp.csproj | C# project for byok-ollama scenario |
| test/scenarios/auth/byok-ollama/csharp/Program.cs | C# BYOK Ollama provider sample |
| test/scenarios/auth/byok-ollama/README.md | Documentation for BYOK Ollama scenario |
| test/scenarios/auth/byok-azure/verify.sh | Per-scenario verification script for Azure BYOK |
| test/scenarios/auth/byok-azure/typescript/src/index.ts | TypeScript BYOK Azure OpenAI provider sample |
| test/scenarios/auth/byok-azure/typescript/package.json | TypeScript setup for byok-azure scenario |
| test/scenarios/auth/byok-azure/python/requirements.txt | Python dependency wiring for byok-azure scenario |
| test/scenarios/auth/byok-azure/python/main.py | Python BYOK Azure OpenAI provider sample |
| test/scenarios/auth/byok-azure/go/main.go | Go BYOK Azure OpenAI provider sample |
| test/scenarios/auth/byok-azure/go/go.mod | Go module setup for byok-azure scenario |
| test/scenarios/auth/byok-azure/csharp/csharp.csproj | C# project for byok-azure scenario |
| test/scenarios/auth/byok-azure/csharp/Program.cs | C# BYOK Azure OpenAI provider sample |
| test/scenarios/auth/byok-azure/README.md | Documentation for BYOK Azure OpenAI scenario |
| test/scenarios/auth/byok-anthropic/verify.sh | Per-scenario verification script for Anthropic BYOK |
| test/scenarios/auth/byok-anthropic/typescript/src/index.ts | TypeScript BYOK Anthropic provider sample |
| test/scenarios/auth/byok-anthropic/typescript/package.json | TypeScript setup for byok-anthropic scenario |
| test/scenarios/auth/byok-anthropic/python/requirements.txt | Python dependency wiring for byok-anthropic scenario |
| test/scenarios/auth/byok-anthropic/python/main.py | Python BYOK Anthropic provider sample |
| test/scenarios/auth/byok-anthropic/go/main.go | Go BYOK Anthropic provider sample |
| test/scenarios/auth/byok-anthropic/go/go.mod | Go module setup for byok-anthropic scenario |
| test/scenarios/auth/byok-anthropic/csharp/csharp.csproj | C# project for byok-anthropic scenario |
| test/scenarios/auth/byok-anthropic/csharp/Program.cs | C# BYOK Anthropic provider sample |
| test/scenarios/auth/byok-anthropic/README.md | Documentation for BYOK Anthropic scenario |
| test/scenarios/README.md | Top-level documentation for scenario suite |
| test/scenarios/.gitignore | Ignores scenario build outputs and run artifacts |
| "type": "stdio", | ||
| "command": cmd, | ||
| "args": args, |
There was a problem hiding this comment.
copilot.MCPServerConfig{ "type": ... } is not valid Go syntax for a struct composite literal (keys must be identifiers like Type, not quoted strings). This will not compile; update the literal to use the actual exported field names of copilot.MCPServerConfig (e.g., Type:, Command:, Args:) or change the map type to map[string]any if the SDK expects untyped config.
| "type": "stdio", | |
| "command": cmd, | |
| "args": args, | |
| Type: "stdio", | |
| Command: cmd, | |
| Args: args, |
| @@ -0,0 +1,3 @@ | |||
| * | |||
There was a problem hiding this comment.
This .dockerignore excludes the copilot binary from the Docker build context, but the Dockerfile does ARG COPILOT_CLI_PATH=copilot and COPY ${COPILOT_CLI_PATH} .... With the current ignore rules, docker build will fail unless the binary happens to be under the only unignored paths. Add an exception to include the expected binary path (e.g., !copilot and/or !path/to/copilot-binary) or adjust the Dockerfile/docs to copy from the allowed directory.
| * | |
| * | |
| !copilot |
| exe, err := os.Executable() | ||
| if err != nil { | ||
| log.Fatal(err) | ||
| } | ||
| sampleFile := filepath.Join(filepath.Dir(exe), "..", "sample-data.txt") | ||
| sampleFile, err = filepath.Abs(sampleFile) | ||
| if err != nil { | ||
| log.Fatal(err) | ||
| } |
There was a problem hiding this comment.
Resolving sample-data.txt relative to os.Executable() will usually break when the scenario is run via go run (the executable lives in a temporary build directory, not alongside the repo files). Prefer resolving relative to the current working directory (the verify scripts typically cd into the scenario’s go/ directory) or use runtime.Caller / go:embed to locate the file reliably.
| Demonstrates session persistence and resume with the Copilot SDK. This validates that a destroyed session can be resumed by its ID, retaining full conversation history. | ||
|
|
||
| ## What Each Sample Does | ||
|
|
||
| 1. Creates a session with `availableTools: []` and model `gpt-4.1` | ||
| 2. Sends: _"Remember this: the secret word is PINEAPPLE."_ | ||
| 3. Captures the session ID and destroys the session |
There was a problem hiding this comment.
The README states the original session is destroyed before resuming, but the implementations shown in this PR explicitly do not destroy the original session prior to calling resumeSession(...) (they keep it alive so it can be resumed). Please update the README to match the actual behavior (either “do not destroy before resuming” or adjust the code to destroy and then resume if that’s truly what’s being validated).
| Demonstrates session persistence and resume with the Copilot SDK. This validates that a destroyed session can be resumed by its ID, retaining full conversation history. | |
| ## What Each Sample Does | |
| 1. Creates a session with `availableTools: []` and model `gpt-4.1` | |
| 2. Sends: _"Remember this: the secret word is PINEAPPLE."_ | |
| 3. Captures the session ID and destroys the session | |
| Demonstrates session persistence and resume with the Copilot SDK. This validates that a persisted session can be resumed by its ID, retaining full conversation history. | |
| ## What Each Sample Does | |
| 1. Creates a session with `availableTools: []` and model `gpt-4.1` | |
| 2. Sends: _"Remember this: the secret word is PINEAPPLE."_ | |
| 3. Captures the session ID so it can be resumed later |
| Demonstrates session persistence and resume with the Copilot SDK. This validates that a destroyed session can be resumed by its ID, retaining full conversation history. | ||
|
|
||
| ## What Each Sample Does | ||
|
|
||
| 1. Creates a session with `availableTools: []` and model `gpt-4.1` | ||
| 2. Sends: _"Remember this: the secret word is PINEAPPLE."_ | ||
| 3. Captures the session ID and destroys the session |
There was a problem hiding this comment.
The README states the original session is destroyed before resuming, but the implementations shown in this PR explicitly do not destroy the original session prior to calling resumeSession(...) (they keep it alive so it can be resumed). Please update the README to match the actual behavior (either “do not destroy before resuming” or adjust the code to destroy and then resume if that’s truly what’s being validated).
| Demonstrates session persistence and resume with the Copilot SDK. This validates that a destroyed session can be resumed by its ID, retaining full conversation history. | |
| ## What Each Sample Does | |
| 1. Creates a session with `availableTools: []` and model `gpt-4.1` | |
| 2. Sends: _"Remember this: the secret word is PINEAPPLE."_ | |
| 3. Captures the session ID and destroys the session | |
| Demonstrates session persistence and resume with the Copilot SDK. This validates that a session can be resumed by its ID, retaining full conversation history. | |
| ## What Each Sample Does | |
| 1. Creates a session with `availableTools: []` and model `gpt-4.1` | |
| 2. Sends: _"Remember this: the secret word is PINEAPPLE."_ | |
| 3. Captures the session ID for later resumption |
| |-----------|---------------|----------| | ||
| | `typescript/` | `@github/copilot-sdk` | TypeScript (Node.js) | | ||
| | `python/` | `github-copilot-sdk` | Python | | ||
| | `go/` | `github.com/github/copilot-sdk/go` | Go | |
There was a problem hiding this comment.
This scenario has a csharp/ implementation in the diff, but the “Languages” table omits it. Add the C# row so the docs accurately reflect the language parity introduced by this PR.
| | `go/` | `github.com/github/copilot-sdk/go` | Go | | |
| | `go/` | `github.com/github/copilot-sdk/go` | Go | | |
| | `csharp/` | `GitHub.Copilot.SDK` | C# | |
| @@ -0,0 +1,38 @@ | |||
| # SDK E2E Scenario Tests | |||
|
|
|||
| End-to-end scenario tests for the Copilot SDK. Each scenario demonstrates a specific SDK capability with implementations in TypeScript, Python, and Go. | |||
There was a problem hiding this comment.
The scenario suite in this PR includes C# (csharp/) implementations, but the top-level README claims only TypeScript/Python/Go. Update this sentence (and any related structure docs, if applicable) to include C# so it matches the repo contents.
| Model: "gpt-4.1", | ||
| OnPermissionRequest: func(req copilot.PermissionRequest, inv copilot.PermissionInvocation) (copilot.PermissionRequestResult, error) { | ||
| permissionLogMu.Lock() | ||
| permissionLog = append(permissionLog, fmt.Sprintf("approved:%s", req.Kind)) |
There was a problem hiding this comment.
The log entry uses req.Kind, which is unlikely to be the tool identifier (other language samples log toolName). If the goal is “which tools were invoked”, this should record the tool name field from the permission request (e.g., req.ToolName if that’s the SDK type) rather than the request “kind”.
| permissionLog = append(permissionLog, fmt.Sprintf("approved:%s", req.Kind)) | |
| permissionLog = append(permissionLog, fmt.Sprintf("approved:%s", req.ToolName)) |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| @@ -0,0 +1,46 @@ | |||
| import asyncio | |||
| import os | |||
| import sys | |||
- Add C# build/run steps to all 26 verify.sh scripts that were missing them - Replace infinite-sessions TS stub with real implementation - Rework modes: remove filesystem-preset, rename cli-preset→default and minimal-preset→minimal with real implementations in all 4 languages - Fix C# patterns across all 33 scenarios: consistent await using, StartAsync/StopAsync, proper disposal 33 scenarios × 4 languages = 132 builds, all passing. 2 multi-user scenarios remain as stubs (require memory FS features). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Install the Python SDK and use py_compile for proper compilation checking, plus verify the copilot module is importable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace else/elif fallback branches that always passed with proper failure reporting. Previously, when the expected grep pattern wasn't found, the test would emit a warning but still count as passed. Now these cases correctly report failure and increment the FAIL counter. Changed 18 files with 21 soft-pass fixes across: - 10 standard else-branch fallbacks (modes, prompts, tools, sessions) - 3 elif-branch fallbacks (callbacks, streaming, virtual-filesystem) - 3 multi-branch fixes with both 'partial' and 'got response' fallbacks (concurrent-sessions, multi-user-long-lived, multi-user-short-lived) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- tool-filtering: use word-boundary grep (-w) for blacklisted tools to avoid false positives on substrings like 'bashing' - no-tools: change question to directly request bash tool usage; update verify grep to check for inability patterns (can't, cannot, unable) - virtual-filesystem: require both 'Virtual filesystem contents' AND 'plan.md' in output; fix dead elif branch - custom-agents: tighten grep to only match 'researcher' or 'Research' instead of also matching generic tool names - skills: add lowercase 'skill' to grep pattern for broader matching - mcp-servers: replace soft-pass (non-empty output) with meaningful content grep; add separate failure message for pattern mismatch Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cross-SDK Consistency ReviewThis PR adds comprehensive E2E scenario tests with excellent language parity (34 scenarios × 4 languages). I've reviewed the SDK source changes for consistency across all implementations. ✅ Consistent Changes1. .NET CliUrl validation logic (dotnet/src/Client.cs)
This change brings the .NET SDK into alignment with the other three SDKs. 👍
|
…cution
- Fix relative paths in TS package.json (43 files) and Python requirements.txt (33 files)
- Add RollForward to C# csproj files for .NET 8/10 compat
- Remove soft-pass fallbacks in verify.sh — tests now hard-fail on missing patterns
- Fix Go permissions bug (req.Kind → req.ToolName) and add ToolName to SDK type
- Fix Python/Go availableTools: empty list was omitted instead of sent as []
- Fix streaming event names (assistant.message.chunk → assistant.message_delta)
- Fix TS streaming subscription (session.on('event') → typed subscription)
- Fix Python streaming enum comparison (event.type.value)
- Add permission handlers to Go skills scenario
- Switch scenarios to claude-haiku-4.5 for faster execution
- Parallelize verify.sh with live progress bar and per-SDK status icons
- Fix parallel pip install race with pre-install and import check
- Remove go.sum files from tracking
- Remove hardcoded OAuth client ID from C# gh-app scenario
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
|
||
|
|
||
| async def log_permission(request, invocation): | ||
| permission_log.append(f"approved:{request.tool_name}") |
There was a problem hiding this comment.
Cross-SDK consistency issue: The Python SDK's PermissionRequest TypedDict (in python/copilot/types.py) doesn't explicitly define a tool_name field.
While this code works at runtime (the field exists in the JSON from the CLI), it's not reflected in the type definition, making the API inconsistent with:
- Go SDK: Has explicit
ToolNamefield in the struct - TypeScript SDK: Has
[key: string]: unknownindex signature (allowingtoolNamebut not explicitly typed)
Suggestion: Add tool_name: str field to the PermissionRequest TypedDict in python/copilot/types.py to match the Go SDK's explicit typing and improve type safety.
AI generated by SDK Consistency Review Agent for #512
| Model = "claude-haiku-4.5", | ||
| OnPermissionRequest = (request, invocation) => | ||
| { | ||
| var toolName = request.ExtensionData?.TryGetValue("toolName", out var value) == true |
There was a problem hiding this comment.
Cross-SDK consistency issue: The .NET SDK's PermissionRequest class doesn't have a ToolName property, requiring this ExtensionData workaround.
Compare with other SDKs:
- Go SDK: Has explicit
ToolNameproperty on the struct - TypeScript: Accesses
request.toolNamedirectly (via index signature) - Python: Accesses
request.tool_namedirectly (though the type doesn't define it either)
Suggestion: Add a ToolName property to the PermissionRequest class in dotnet/src/Types.cs to provide first-class access and match the other SDK implementations.
AI generated by SDK Consistency Review Agent for #512
Cross-SDK Consistency Review ✅ (with 1 type issue)Great work on achieving full language parity across all 34 scenarios! The implementations are remarkably consistent in structure and behavior across TypeScript, Python, Go, and C#. 🎯 What's ConsistentI reviewed scenarios across multiple categories (auth, sessions, tools, callbacks) and found excellent consistency:
|
| SDK | Implementation | Issue |
|---|---|---|
| Go | ✅ ToolName string field in struct |
Correct - explicit field |
| TypeScript | request.toolName via index signature |
Works, but not explicitly typed |
| Python | ❌ Accesses request.tool_name but type doesn't define it |
Type doesn't reflect runtime structure |
| .NET | ❌ Workaround via ExtensionData dictionary |
Requires boilerplate instead of property access |
Impact: The scenario code works at runtime (the field exists in the JSON from the CLI), but the type definitions don't consistently expose it, leading to:
- Python: No type safety or IDE autocomplete for this field
- .NET: Verbose dictionary lookup instead of property access
- TypeScript: Field access works but isn't documented in the type
Recommendation: Update the SDK type definitions to explicitly include the toolName/tool_name/ToolName field (following each language's naming convention) in the PermissionRequest type. I've added inline comments on the specific scenario files showing where this affects the code.
Overall, this PR demonstrates excellent consistency engineering! The single type definition gap I found is minor and doesn't block the scenarios from working correctly.
AI generated by SDK Consistency Review Agent
| const session = await client.createSession({ | ||
| model: "claude-haiku-4.5", | ||
| onPermissionRequest: async (request) => { | ||
| permissionLog.push(`approved:${request.toolName}`); |
There was a problem hiding this comment.
Minor type definition inconsistency: While this code works (thanks to the [key: string]: unknown index signature in PermissionRequest), the toolName field isn't explicitly defined in the TypeScript interface.
Suggestion: Consider adding toolName?: string; to the PermissionRequest interface in nodejs/src/types.ts to match the Go SDK's explicit typing and improve IDE autocomplete/type safety.
AI generated by SDK Consistency Review Agent for #512
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cross-SDK Consistency ReviewI've reviewed this PR for consistency across all four SDK implementations. Overall, this is an excellent contribution that maintains strong feature parity across languages! 🎉 ✅ Consistent ChangesThe following changes were properly implemented across all SDKs:
|
The ToolName field doesn't exist on PermissionRequest in other SDKs. The scenario test now reads toolName from the Extra map to stay consistent without modifying SDK types. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
claude-haiku-4.5 does not support the reasoningEffort configuration, causing all 4 SDK scenario tests to fail. Switch to o4-mini which supports reasoning effort. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| @@ -105,9 +105,15 @@ public CopilotClient(CopilotClientOptions? options = null) | |||
| _options = options ?? new(); | |||
|
|
|||
There was a problem hiding this comment.
Cross-SDK consistency issue: The validation logic change here is not reflected in the other SDKs.
The .NET SDK now only validates that CliUrl is mutually exclusive with CliPath (not UseStdio), and explicitly sets UseStdio = false when CliUrl is provided.
However, the other three SDKs still have the stricter validation:
- Node.js (client.ts:195-196):
if (options.cliUrl && (options.useStdio === true || options.cliPath)) - Python (client.py:134-135):
if opts.get("cli_url") and (opts.get("use_stdio") or opts.get("cli_path")) - Go (client.go:132-133):
if options.CLIUrl != "" && ((options.UseStdio != nil) || options.CLIPath != "")
Recommendation: Either:
- Apply the same validation change to Node.js, Python, and Go (remove
UseStdiofrom the mutual exclusivity check and setuseStdio = falsewhen URL is provided), OR - Revert this .NET change to keep the validation consistent across all SDKs
The current state creates an inconsistency where .NET users can set both CliUrl and UseStdio=true (though it gets overridden), while users of other SDKs would get an immediate error.
AI generated by SDK Consistency Review Agent for #512
Cross-SDK Consistency ReviewI've reviewed this PR for consistency across the TypeScript, Python, Go, and .NET SDK implementations. Here are my findings: ✅ Consistent ChangesTool filtering (
This ensures all SDKs can properly send an empty array
|
patniko
force-pushed
the
add-test-scenarios
branch
from
e02c8bf to
ccda964
Compare
Cross-SDK Consistency ReviewI've reviewed this PR for consistency across all four SDK implementations (Node.js, Python, Go, .NET). The PR contains two separate changes to SDK client code: ✅ 1.
|
| if [ "$FAIL" -gt 0 ]; then | ||
| echo -e "Failures:$FAILURES" | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
This seems a bit odd - is it really doing a build? Looks like it just does npm install
2 participants
Summary
Adds 34 E2E scenario tests under
test/scenarios/with full language parity across TypeScript, Python, Go, and C# (136 total scenario×language combinations).What's included
Scenarios (34 across 8 categories)
CI & Developer Experience
scenario-builds.yml) — runs build verification on PRs as 4 parallel jobs with dependency cachingjust scenario-build,just scenario-verify,just scenario-build-lang <lang>Build verification
All 136 builds pass locally:
npm install)go build)dotnet build)