* Restructure agent host E2E tests into conformance and parity tiers
The E2E suite ran every test once per provider, so 52 provider-invariant
tests were executed three times on each of three operating systems for a
single meaningful assertion. Split the suite into two tiers:
- conformance: provider-invariant Agent Host Protocol behavior, run once
against a single reference provider
- parity: behavior that must be verified separately for Claude, Copilot,
and Codex
Add an IAgentHostTarget seam so the suite can launch a non-VS Code
Agent Host Protocol implementation, keeping the tests external to the
implementation under test.
Track protocol-surface coverage (commands, notifications, and action
types observed on the wire) alongside the existing line coverage, and
check the stats in so gaps are visible in review.
Freeze the protocol/ suite: it side-loads a mock agent into the
production server, so it cannot be run against an alternate
implementation. Record the migration backlog in the E2E README.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Record request-assertion and line-ending gaps in E2E known issues
Document two structural gaps found while reviewing the record/replay
design:
- The recorded model request is normalized on write but never read back,
so replay cannot detect regressions in prompt assembly, history
retention, or attachment marshalling. Records the projected-assertion
approach and why tool result text must be elided from it.
- Snapshot normalization does not handle line endings, so any snapshot
carrying literal text can fail on Windows for reasons unrelated to the
behavior under test.
Also note the portable-command guidance (`node -e` / a seeded script) for
scenarios that genuinely need to run a command.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback on E2E coverage tooling
- Derive the conformance suite title from the provider config instead of
passing it separately, matching defineAgentHostE2ETests and removing a
second source of truth for the suite name.
- Warn once when protocol-surface observations cannot be written. The
write is still non-fatal, but a silent failure previously surfaced much
later as the coverage script reporting a missing observation file.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Gate the Claude side-chat context test on a server-tool wiring race
`side chat receives bounded source context without copied history` fails
intermittently on Claude with `Server not found: host`, raised by the
Claude CLI when a replayed turn calls a server tool before the host's
server-tool MCP server is registered for that session.
Separating the conformance and parity tiers concentrated the Claude
parity suite into consecutive model-backed turns where interleaved
host-only tests previously spaced them out, which loses this race about
half the time. Measured at ~4 failures in 8 full-suite runs, against 0 in
6 runs before the split; the test passes in isolation and with a fresh
server per test, so it is a materialization race rather than replay or
shared-server state.
Gate it behind `sideChatServerToolWiringUnstable` and record the
measurements in KNOWN_ISSUES.md, including a note not to re-record the
capture to make it pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Link the side-chat race to its tracking issue
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Drop the side-chat gate now that the underlying race is fixed
#327560 fixed the root cause: the Claude session published the host's own
in-process `host` and `client` MCP bridges into session-scoped state, so a
peer or side chat whose query had not yet reported `host` tried to toggle
it and the CLI answered `Server not found: host`.
That is the same failure this branch quarantined, so remove the
`sideChatServerToolWiringUnstable` gate and its known-issue entry rather
than carrying a stale workaround. Verified with the gate removed: 4 clean
Claude runs and 2 clean full-suite runs (150 passing, 0 failing).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reorganize Agent Host integration tests
Separate protocol, provider E2E, mocked-LLM, and direct SDK test families, and split shared provider scenarios into focused suites.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Agent Host test review feedback
Keep coverage scope metadata and replay fixture documentation aligned with the reorganized suites.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Limit Agent Host coverage to provider E2E tests
Exclude mock-agent and mocked-LLM suites so the report measures only the real server and bundled provider stacks with replayed model traffic.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Clarify Agent Host E2E test boundaries
Move synthetic-LLM suites into provider integration, flatten E2E captures, and simplify the checked-in coverage summary path without changing tests or fixture contents.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
