Files
43ddfbfb38 Make UI scenario validation work against an installed build, without an MCP server (#331674)
* Fix automation MCP launches against an installed build

getApplication re-applied the from-source environment on every launch, so
--build runs were also marked VSCODE_DEV=1. A packaged build then behaves as
if it were running from a checkout and never opens a window, and the launch fails
with Timeout 60000ms exceeded while waiting for event ''window''. That
environment is already resolved at module load, and only for the from-source
path, so drop the duplicate.

The launcher also never passed an extensions directory, so a --build run loaded
the user's installed extensions: those change the product under test and their
logs are copied into the evidence bundle. Point it at an isolated directory under
the existing test data path, matching the smoke test runner.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: adb443eb-11e5-40a1-8608-7f593fa79485

* Stop downloading a stable build the MCP server never uses

\setup\ downloaded the previous stable release on every \--build\ launch, and
the test data directory it downloads into is wiped at startup, so the cost was
paid on every run: 335 MB and ~36s before VS Code even starts.

The result was only ever assigned to \opts['stable-build']\, which nothing in
this server reads. Migration tests belong to the smoke test runner, which has its
own copy of this logic. Remove the download along with the now-unused
\--stable-build\ option.

A full \--build\ evidence run drops from about 90s to 19s.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: adb443eb-11e5-40a1-8608-7f593fa79485

* Run UI validation scenarios without configuring an MCP server

The ui-scenario-validation skill could only be used by first registering
test/mcp as an MCP server and then driving the evidence tools by hand. That is a
lot of setup for a one-off reproduction, and it puts the capability out of reach
of a plain request like ''reproduce this issue and record it''.

Add runScenario, which takes a scenario file and runs it end to end:

    node test/mcp/out/runScenario.js <scenario.js> --build <app-root>

A scenario is plain JavaScript that exports an id, a title and a list of steps.
Each step gets the Application, the Workbench helpers and the Playwright page,
returns a string describing how it was validated, throws to fail, or calls
skip(reason) when a precondition is unavailable. The runner launches VS Code with
video and tracing, captures a screenshot at every step boundary, stops at the
first failed or skipped step, writes the report and renders the chapter titles
onto the recording.

Clean capture is enabled by default, so the recording shows unmodified UI and the
step titles are added afterwards.

A skipped step reports the run as aborted rather than passed: the scenario did
not validate, so calling it a pass would overstate the result.

Rewrite the skill around this flow. The MCP server remains available for
interactive exploration, where inspecting the UI before choosing an assertion is
worth the setup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: adb443eb-11e5-40a1-8608-7f593fa79485

* Caption the recording instead of cutting chapter cards into it

Step titles were shown on full-screen cards spliced between segments, which
stretched the timeline and told the viewer nothing about what the step actually
checked. The validation detail was meant to appear on those cards but never did:
it was read from the opening capture, which only records that a step started, so
the field was always empty.

Draw a caption band under the frame instead. ffmpeg pads the canvas and the text
is drawn into the added strip, so the recording keeps its original length and no
recorded pixel is covered - the status bar and the bottom of the workbench stay
readable. Each caption shows the step number and id, its status, the title, and
the validation detail from the closing capture, coloured green, red or amber to
match. The run outcome sits in the corner for the whole video.

This also collapses the render to a single ffmpeg pass: the previous version
trimmed one segment per step, generated a card per step and concatenated them.

Give a CommonJS scenario a .cjs extension. This package is an ES module
package, so a scenario saved as .js inside the repository is treated as ESM and
a CommonJS one fails to load - which is exactly what the skill told people to do.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: adb443eb-11e5-40a1-8608-7f593fa79485

* Rename the skill to validate-ui-scenario

Workspace skills are surfaced as slash commands named after the skill, so the
name is what a user types. "ui-scenario-validation" reads as a topic; the verb
form matches how it is invoked:

    /validate-ui-scenario reproduce <issue url> against my installed Insiders

Update the example to that phrasing, and state that a reproduction which fails at
the expected step is a successful reproduction rather than a broken scenario.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: adb443eb-11e5-40a1-8608-7f593fa79485

* Put the caption band above the recorded frame

The band reads before the frame it describes, and it keeps the eye near the
toolbar and editor where the action happens rather than at the far edge of the
window.

The canvas is still padded rather than overlaid, so no recorded pixel is hidden.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: adb443eb-11e5-40a1-8608-7f593fa79485

* Reject per-run arguments that escape the isolated profile

Per-run `extraArgs` are appended after the generated arguments, and VS Code keeps
the last value of a repeated string option, so a caller-supplied
`--extensions-dir` would replace the isolated directory and let real user
extensions and their logs back into the recording. The same applies to
`--user-data-dir` on the restart path, which had no guard at all.

Move the check into a shared helper and apply it to both the launch and restart
argument paths.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: adb443eb-11e5-40a1-8608-7f593fa79485

* Unwrap a default export without the `in` operator

`local/code-no-in-operator` rejects `'default' in loaded`. Reading the optional
property directly is equivalent here and simpler: a CommonJS scenario has no
`default`, and an ES module namespace carries the scenario on it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: adb443eb-11e5-40a1-8608-7f593fa79485

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: adb443eb-11e5-40a1-8608-7f593fa79485
2026-08-19 17:03:39 -07:00
..
2026-07-14 12:34:04 +02:00

Code - OSS Development MCP Server

This directory contains a Model Context Protocol (MCP) server that provides VS Code automation capabilities for Code - OSS development and testing. The MCP server exposes Code - OSS's testing infrastructure through a standardized interface, allowing AI assistants and other tools to interact with VS Code programmatically.

What is MCP?

The Model Context Protocol (MCP) is an open standard that enables AI assistants to securely connect to external data sources and tools. This MCP server specifically provides VS Code automation capabilities, making it possible for AI assistants to:

  • Start and stop VS Code instances
  • Interact with editors, terminals, and UI elements
  • Run commands and keybindings
  • Navigate the explorer, search, debug, and other viewlets
  • Manage extensions, settings, and keybindings
  • Work with notebooks and chat features

Quick Start - Stdio

Firstly, make sure you install all dependencies (npm i) at the root of the repo.

Then, open the Command Palette and run:

MCP: List Servers → vscode-automation-mcp → Start Server

or open mcp.json and start it from there.

That's it! It should automatically compile everything needed.

Arguments

Open the mcp.json and modify the args:

  • ["run", "start-stdio"]: opens Electron window
  • ["run", "start-stdio", "--", "--web"]: opens a Chromium window
  • ["run", "start-stdio", "--", "--web", "--headless"]: opens a headless window

NOTE: --web requires running npm run install-playwright from root

Debugging the server

You can modify the mcp.json to debug the server:

"vscode-automation-mcp": {
	"type": "stdio",
	"command": "node",
	"args": ["./out/stdio.js"],
	"cwd": "${workspaceFolder}/test/mcp",
	"dev": {
		"watch": "test/mcp/**/*.ts",
		"debug": {
			"type": "node"
		}
	}
}

What the Server Provides

The MCP server exposes a comprehensive set of VS Code automation tools through the MCP protocol:

Application Management

  • Start, stop, and restart VS Code instances
  • Open workspaces and folders
  • Record scenario evidence with step overlays, screenshots, video, traces, and an HTML report

Editor Tools

  • Open, close, and navigate files
  • Get and set editor content
  • Manage selections and cursors

Terminal Tools

  • Create and manage terminal instances
  • Send commands to terminals
  • Read terminal output

Debug Tools

  • Start and stop debug sessions
  • Manage breakpoints
  • Step through code

Search Tools

  • Search for files and text
  • Navigate search results

Extension Tools

  • Install and manage extensions
  • View extension information

UI Interaction

  • Quick access and command palette
  • Explorer and activity bar
  • Source control management
  • Status bar interactions
  • Problems panel
  • Settings and keybindings editors
  • Notebook support
  • Chat features

Scenario evidence

Start evidence capture before starting VS Code so Playwright enables video recording:

  1. Call vscode_automation_evidence_start, including any required pre-launch userSettings.
  2. Call vscode_automation_evidence_step with started before each action.
  3. Perform and validate the action with the relevant automation tools.
  4. Call vscode_automation_evidence_step with passed, failed, or skipped.
  5. Call vscode_automation_evidence_finish.

Artifacts are written to .build/vscode-playwright-mcp/evidence/<run-id>/. Each evidence run uses an isolated user profile with in-memory secret storage and records every visited Playwright page as a separate video.

Development

Manual Setup (Advanced)

If you prefer to run the server manually:

# Navigate to the MCP directory
cd test/mcp

# Install dependencies
npm install

# Compile TypeScript
npm run compile # or watch

# Start the server
npm start

Project Structure

test/mcp/
├── src/
│   ├── stdio.ts             # Entry point for stdio transport
│   ├── automation.ts        # MCP server with automation tools
│   ├── application.ts       # VS Code application lifecycle management
│   ├── options.ts           # Command-line options parsing
│   ├── utils.ts             # Utility functions
│   └── automationTools/     # Tool implementations organized by feature
│       ├── index.ts         # Tool registration
│       ├── core.ts          # Core application tools
│       ├── editor.ts        # Editor tools
│       ├── terminal.ts      # Terminal tools
│       ├── debug.ts         # Debug tools
│       └── ...              # Other feature-specific tools
├── package.json             # Dependencies and scripts
├── tsconfig.json            # TypeScript configuration
└── README.md                # This file

Architecture

The server uses a simple architecture:

  • stdio.ts - Entry point that creates the MCP server and connects via stdio transport
  • automation.ts - Creates the MCP server and registers all automation tools
  • application.ts - Manages VS Code application lifecycle (start, stop, restart)
  • automationTools/ - Modular tool implementations organized by VS Code feature area

Troubleshooting

Server Won't Start

  • Ensure Code - OSS has been built and run at least once (via F5 or code.sh)
  • Verify all dependencies are installed with npm install

Automation Issues

  • Ensure Code - OSS has been built and run at least once (via F5 or code.sh)
  • Check the server logs for errors
  • Verify the workspace path is correct

Contributing

This MCP server is part of the Code - OSS development infrastructure. When making changes:

  1. Follow the existing TypeScript and coding conventions
  2. Test with multiple MCP clients if possible
  3. Update this README if adding new capabilities
  4. Ensure proper error handling and logging

License

This project is licensed under the MIT License - see the top-level project's license file for details.