* agentHost: run E2E suites in parallel
Run the deterministic conformance and provider entrypoints concurrently while preserving serial server reuse within each suite. Integrate the parallel runner into coverage and full integration runs.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* agentHost: harden parallel E2E runner
Preserve Windows argument boundaries through a PowerShell wrapper and fail coverage runs when a worker does not emit protocol-surface observations.
(Written by Copilot)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* agentHost: stabilize terminal clear E2E test
Wait for a marker that appears only in command output, not in the shell's echoed input, before clearing terminal state.
(Written by Copilot)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* agentHost: replace gated SDK tests
Run the unique SDK compatibility checks tokenlessly in normal CI and remove the obsolete real-SDK gate and helper.\n\n(Written by Copilot)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* agentHost: fix SDK tests on Windows
Launch direct SDK integration tests with the same sanitized Node-mode environment used by the production Copilot agent.\n\n(Written by Copilot)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restructure agent host E2E tests into conformance and parity tiers
The E2E suite ran every test once per provider, so 52 provider-invariant
tests were executed three times on each of three operating systems for a
single meaningful assertion. Split the suite into two tiers:
- conformance: provider-invariant Agent Host Protocol behavior, run once
against a single reference provider
- parity: behavior that must be verified separately for Claude, Copilot,
and Codex
Add an IAgentHostTarget seam so the suite can launch a non-VS Code
Agent Host Protocol implementation, keeping the tests external to the
implementation under test.
Track protocol-surface coverage (commands, notifications, and action
types observed on the wire) alongside the existing line coverage, and
check the stats in so gaps are visible in review.
Freeze the protocol/ suite: it side-loads a mock agent into the
production server, so it cannot be run against an alternate
implementation. Record the migration backlog in the E2E README.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Record request-assertion and line-ending gaps in E2E known issues
Document two structural gaps found while reviewing the record/replay
design:
- The recorded model request is normalized on write but never read back,
so replay cannot detect regressions in prompt assembly, history
retention, or attachment marshalling. Records the projected-assertion
approach and why tool result text must be elided from it.
- Snapshot normalization does not handle line endings, so any snapshot
carrying literal text can fail on Windows for reasons unrelated to the
behavior under test.
Also note the portable-command guidance (`node -e` / a seeded script) for
scenarios that genuinely need to run a command.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback on E2E coverage tooling
- Derive the conformance suite title from the provider config instead of
passing it separately, matching defineAgentHostE2ETests and removing a
second source of truth for the suite name.
- Warn once when protocol-surface observations cannot be written. The
write is still non-fatal, but a silent failure previously surfaced much
later as the coverage script reporting a missing observation file.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Gate the Claude side-chat context test on a server-tool wiring race
`side chat receives bounded source context without copied history` fails
intermittently on Claude with `Server not found: host`, raised by the
Claude CLI when a replayed turn calls a server tool before the host's
server-tool MCP server is registered for that session.
Separating the conformance and parity tiers concentrated the Claude
parity suite into consecutive model-backed turns where interleaved
host-only tests previously spaced them out, which loses this race about
half the time. Measured at ~4 failures in 8 full-suite runs, against 0 in
6 runs before the split; the test passes in isolation and with a fresh
server per test, so it is a materialization race rather than replay or
shared-server state.
Gate it behind `sideChatServerToolWiringUnstable` and record the
measurements in KNOWN_ISSUES.md, including a note not to re-record the
capture to make it pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Link the side-chat race to its tracking issue
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Drop the side-chat gate now that the underlying race is fixed
#327560 fixed the root cause: the Claude session published the host's own
in-process `host` and `client` MCP bridges into session-scoped state, so a
peer or side chat whose query had not yet reported `host` tried to toggle
it and the CLI answered `Server not found: host`.
That is the same failure this branch quarantined, so remove the
`sideChatServerToolWiringUnstable` gate and its known-issue entry rather
than carrying a stale workaround. Verified with the gate removed: 4 clean
Claude runs and 2 clean full-suite runs (150 passing, 0 failing).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reorganize Agent Host integration tests
Separate protocol, provider E2E, mocked-LLM, and direct SDK test families, and split shared provider scenarios into focused suites.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Agent Host test review feedback
Keep coverage scope metadata and replay fixture documentation aligned with the reorganized suites.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Limit Agent Host coverage to provider E2E tests
Exclude mock-agent and mocked-LLM suites so the report measures only the real server and bundled provider stacks with replayed model traffic.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Clarify Agent Host E2E test boundaries
Move synthetic-LLM suites into provider integration, flatten E2E captures, and simplify the checked-in coverage summary path without changing tests or fixture contents.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
