mirror of
https://github.com/microsoft/vscode.git
synced 2026-08-14 17:54:53 +01:00
018354116a
* Restructure agent host E2E tests into conformance and parity tiers The E2E suite ran every test once per provider, so 52 provider-invariant tests were executed three times on each of three operating systems for a single meaningful assertion. Split the suite into two tiers: - conformance: provider-invariant Agent Host Protocol behavior, run once against a single reference provider - parity: behavior that must be verified separately for Claude, Copilot, and Codex Add an IAgentHostTarget seam so the suite can launch a non-VS Code Agent Host Protocol implementation, keeping the tests external to the implementation under test. Track protocol-surface coverage (commands, notifications, and action types observed on the wire) alongside the existing line coverage, and check the stats in so gaps are visible in review. Freeze the protocol/ suite: it side-loads a mock agent into the production server, so it cannot be run against an alternate implementation. Record the migration backlog in the E2E README. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Record request-assertion and line-ending gaps in E2E known issues Document two structural gaps found while reviewing the record/replay design: - The recorded model request is normalized on write but never read back, so replay cannot detect regressions in prompt assembly, history retention, or attachment marshalling. Records the projected-assertion approach and why tool result text must be elided from it. - Snapshot normalization does not handle line endings, so any snapshot carrying literal text can fail on Windows for reasons unrelated to the behavior under test. Also note the portable-command guidance (`node -e` / a seeded script) for scenarios that genuinely need to run a command. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback on E2E coverage tooling - Derive the conformance suite title from the provider config instead of passing it separately, matching defineAgentHostE2ETests and removing a second source of truth for the suite name. - Warn once when protocol-surface observations cannot be written. The write is still non-fatal, but a silent failure previously surfaced much later as the coverage script reporting a missing observation file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Gate the Claude side-chat context test on a server-tool wiring race `side chat receives bounded source context without copied history` fails intermittently on Claude with `Server not found: host`, raised by the Claude CLI when a replayed turn calls a server tool before the host's server-tool MCP server is registered for that session. Separating the conformance and parity tiers concentrated the Claude parity suite into consecutive model-backed turns where interleaved host-only tests previously spaced them out, which loses this race about half the time. Measured at ~4 failures in 8 full-suite runs, against 0 in 6 runs before the split; the test passes in isolation and with a fresh server per test, so it is a materialization race rather than replay or shared-server state. Gate it behind `sideChatServerToolWiringUnstable` and record the measurements in KNOWN_ISSUES.md, including a note not to re-record the capture to make it pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Link the side-chat race to its tracking issue Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Drop the side-chat gate now that the underlying race is fixed #327560 fixed the root cause: the Claude session published the host's own in-process `host` and `client` MCP bridges into session-scoped state, so a peer or side chat whose query had not yet reported `host` tried to toggle it and the CLI answered `Server not found: host`. That is the same failure this branch quarantined, so remove the `sideChatServerToolWiringUnstable` gate and its known-issue entry rather than carrying a stale workaround. Verified with the gate removed: 4 clean Claude runs and 2 clean full-suite runs (150 passing, 0 failing). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
52 lines
1.0 KiB
Plaintext
52 lines
1.0 KiB
Plaintext
.DS_Store
|
|
.cache
|
|
npm-debug.log
|
|
Thumbs.db
|
|
node_modules/
|
|
.tmp/
|
|
.build/
|
|
/extensionsCG/
|
|
.vscode/extensions/**/out/
|
|
extensions/**/dist/
|
|
src/vs/base/browser/ui/codicons/codicon/codicon.ttf
|
|
/out*/
|
|
/extensions/**/out/
|
|
build/node_modules
|
|
build/darwin/.dmgbuild
|
|
coverage/
|
|
!/src/vs/platform/agentHost/test/node/e2e/coverage/
|
|
test_data/
|
|
test-results/
|
|
test-results.xml
|
|
vscode.lsif
|
|
vscode.db
|
|
/.profile-oss
|
|
/cli/target
|
|
/cli/openssl
|
|
product.overrides.json
|
|
*.snap.actual
|
|
*.tsbuildinfo
|
|
.eslintcache
|
|
.vscode-test
|
|
.chat-simulation-data
|
|
vscode-telemetry-docs/
|
|
test-output.json
|
|
test/componentFixtures/.screenshots/*
|
|
!test/componentFixtures/.screenshots/baseline/
|
|
dist
|
|
.playwright-cli
|
|
.playwright-mcp
|
|
.claude/
|
|
.agents/agents/*.local.md
|
|
.github/agents/*.local.md
|
|
.agents/agents/*.local.agent.md
|
|
.github/agents/*.local.agent.md
|
|
.agents/hooks/*.local.json
|
|
.github/hooks/*.local.json
|
|
.agents/instructions/*.local.instructions.md
|
|
.github/instructions/*.local.instructions.md
|
|
.agents/prompts/*.local.prompt.md
|
|
.github/prompts/*.local.prompt.md
|
|
.agents/skills/.local/
|
|
.github/skills/.local/
|