mirror of
https://github.com/microsoft/vscode.git
synced 2026-07-10 16:49:36 +01:00
Add performance tests (#309700)
This commit is contained in:
Paul
GitHub
@@ -0,0 +1,265 @@
|
||||
---
|
||||
name: chat-perf
|
||||
description: Run chat perf benchmarks and memory leak checks against the local dev build or any published VS Code version. Use when investigating chat rendering regressions, validating perf-sensitive changes to chat UI, or checking for memory leaks in the chat response pipeline.
|
||||
---
|
||||
|
||||
# Chat Performance Testing
|
||||
|
||||
## When to use
|
||||
|
||||
- Before/after modifying chat rendering code (`chatListRenderer.ts`, `chatInputPart.ts`, markdown rendering)
|
||||
- When changing the streaming response pipeline or SSE processing
|
||||
- When modifying disposable/lifecycle patterns in chat components
|
||||
- To compare performance between two VS Code releases
|
||||
- In CI to gate PRs that touch chat UI code
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# Run perf regression test (compares local dev build vs VS Code 1.115.0):
|
||||
npm run perf:chat -- --scenario text-only --runs 3
|
||||
|
||||
# Run all scenarios with no baseline (just measure):
|
||||
npm run perf:chat -- --no-baseline --runs 3
|
||||
|
||||
# Compare two local builds (apples-to-apples):
|
||||
npm run perf:chat -- --build /path/to/build-A --baseline-build /path/to/build-B --runs 5
|
||||
|
||||
# Build a local production package and compare against a release:
|
||||
npm run perf:chat -- --production-build --baseline-build 1.115.0 --runs 5
|
||||
|
||||
# Run memory leak check (10 messages in one session):
|
||||
npm run perf:chat-leak
|
||||
|
||||
# Run leak check with more messages for accuracy:
|
||||
npm run perf:chat-leak -- --messages 20 --verbose
|
||||
```
|
||||
|
||||
## Perf regression test
|
||||
|
||||
**Script:** `scripts/chat-simulation/test-chat-perf-regression.js`
|
||||
**npm:** `npm run perf:chat`
|
||||
|
||||
Launches VS Code via Playwright Electron, opens the chat panel, sends a message with a mock LLM response, and measures timing, layout, and rendering metrics. By default, downloads VS Code 1.115.0 as a baseline, benchmarks it, then benchmarks the local dev build and compares.
|
||||
|
||||
### Key flags
|
||||
|
||||
| Flag | Default | Description |
|
||||
|---|---|---|
|
||||
| `--runs <n>` | `5` | Runs per scenario. More = more stable. Use 5+ for CI. |
|
||||
| `--scenario <id>` / `-s` | all | Scenario to test (repeatable). See `common/perf-scenarios.js`. |
|
||||
| `--build <path\|ver>` / `-b` | local dev | Build to test. Accepts path or version (`1.110.0`, `insiders`, commit hash). |
|
||||
| `--baseline <path>` | — | Compare against a previously saved baseline JSON file. |
|
||||
| `--baseline-build <path\|ver>` | `1.115.0` | Version or local path to benchmark as baseline. |
|
||||
| `--no-baseline` | — | Skip baseline comparison entirely. |
|
||||
| `--save-baseline` | — | Save results as the new baseline (requires `--baseline <path>`). |
|
||||
| `--resume <path>` | — | Resume a previous run, adding more iterations to increase confidence. |
|
||||
| `--threshold <frac>` | `0.2` | Regression threshold (0.2 = flag if 20% slower). |
|
||||
| `--production-build` | — | Build a local bundled package via `gulp vscode` for comparison against a release baseline. |
|
||||
| `--no-cache` | — | Ignore cached baseline data, always run fresh. |
|
||||
| `--force` | — | Skip build mode mismatch confirmation prompt. |
|
||||
| `--ci` | — | CI mode: write Markdown summary to `ci-summary.md` (implies `--no-cache`). |
|
||||
| `--setting <k=v>` | — | Set a VS Code setting override for all builds (repeatable). |
|
||||
| `--test-setting <k=v>` | — | Set a VS Code setting override for the test build only. |
|
||||
| `--baseline-setting <k=v>` | — | Set a VS Code setting override for the baseline build only. |
|
||||
| `--verbose` | — | Print per-run details including response content. |
|
||||
|
||||
### Comparing two remote builds
|
||||
|
||||
```bash
|
||||
# Compare 1.110.0 against 1.115.0 (no local build needed):
|
||||
npm run perf:chat -- --build 1.110.0 --baseline-build 1.115.0 --runs 5
|
||||
```
|
||||
|
||||
### Comparing two local builds
|
||||
|
||||
Both `--build` and `--baseline-build` accept local paths to VS Code executables. This enables apples-to-apples comparisons between any two builds:
|
||||
|
||||
```bash
|
||||
# Compare two dev builds (e.g. feature branch vs main):
|
||||
npm run perf:chat -- \
|
||||
--build .build/electron/Code\ -\ OSS.app/Contents/MacOS/Code\ -\ OSS \
|
||||
--baseline-build /path/to/other/Code\ -\ OSS.app/Contents/MacOS/Code\ -\ OSS \
|
||||
--runs 5
|
||||
|
||||
# Compare two production builds:
|
||||
npm run perf:chat -- \
|
||||
--build ../VSCode-darwin-arm64-feature/Code\ -\ OSS.app/Contents/MacOS/Code\ -\ OSS \
|
||||
--baseline-build ../VSCode-darwin-arm64-main/Code\ -\ OSS.app/Contents/MacOS/Code\ -\ OSS \
|
||||
--runs 5
|
||||
```
|
||||
|
||||
Local path baselines are never cached (the build may change between runs). Version string baselines are cached for reuse.
|
||||
|
||||
### Build modes and mismatch detection
|
||||
|
||||
The tool classifies builds into three modes based on the executable path:
|
||||
|
||||
| Mode | Source | Characteristics |
|
||||
|---|---|---|
|
||||
| `dev` | `.build/electron/` (local dev) | Unbundled sources, `VSCODE_DEV=1`, `NODE_ENV=development`. Higher memory and startup overhead. |
|
||||
| `production` | `../VSCode-<platform>-<arch>/` (from `gulp vscode`) | Bundled JS, no dev flags. Matches release characteristics but uses local source. |
|
||||
| `release` | `.vscode-test/` (downloaded via `@vscode/test-electron`) | Official published build. |
|
||||
|
||||
When test and baseline builds have different modes (e.g. dev vs release), the tool shows a warning and prompts for confirmation. Use `--force` or `--ci` to skip the prompt.
|
||||
|
||||
Using `--production-build` builds a local bundled package via `gulp vscode` for fair comparison against a release baseline. This eliminates dev-mode overhead while still testing your local changes.
|
||||
|
||||
```bash
|
||||
# Production build vs release baseline (fair comparison):
|
||||
npm run perf:chat -- --production-build --baseline-build 1.115.0 --runs 5
|
||||
```
|
||||
|
||||
### Settings overrides
|
||||
|
||||
Use `--setting`, `--test-setting`, and `--baseline-setting` to inject VS Code settings into the launched instance. This is useful for A/B testing experimental features:
|
||||
|
||||
```bash
|
||||
# Enable a feature for the test build only:
|
||||
npm run perf:chat -- --test-setting chat.experimental.incrementalRendering.enabled=true --runs 3
|
||||
|
||||
# Compare two builds with different settings:
|
||||
npm run perf:chat -- \
|
||||
--baseline-build "../vscode2/.build/electron/Code - OSS.app/Contents/MacOS/Code - OSS" \
|
||||
--baseline-setting chat.experimental.incrementalRendering.enabled=true \
|
||||
--test-setting chat.experimental.incrementalRendering.enabled=false \
|
||||
--runs 3
|
||||
|
||||
# Set a value for both builds:
|
||||
npm run perf:chat -- --setting chat.mcp.enabled=false --runs 3
|
||||
```
|
||||
|
||||
Precedence: `--test-setting` / `--baseline-setting` override `--setting` for the same key. Values are auto-parsed: `true`/`false` become booleans, numbers become numbers, everything else stays a string.
|
||||
|
||||
### Resuming a run for more confidence
|
||||
|
||||
When results exceed the threshold but aren't statistically significant, the tool prints a `--resume` hint. Use it to add more iterations to an existing run:
|
||||
|
||||
```bash
|
||||
# Initial run with 3 iterations — may be inconclusive:
|
||||
npm run perf:chat -- --scenario text-only --runs 3
|
||||
|
||||
# Add 3 more runs to the same results file (both test + baseline):
|
||||
npm run perf:chat -- --resume .chat-simulation-data/2026-04-14T02-15-14/results.json --runs 3
|
||||
|
||||
# Keep adding until confidence is reached:
|
||||
npm run perf:chat -- --resume .chat-simulation-data/2026-04-14T02-15-14/results.json --runs 5
|
||||
```
|
||||
|
||||
`--resume` loads the previous `results.json` and its associated `baseline-*.json`, runs N more iterations for both builds, merges rawRuns, recomputes stats, and re-runs the comparison. The updated files are written back in-place. You can resume multiple times — samples accumulate.
|
||||
|
||||
### Statistical significance
|
||||
|
||||
Regression detection uses **Welch's t-test** to avoid false positives from noisy measurements. A metric is only flagged as `REGRESSION` when it both exceeds the threshold AND is statistically significant (p < 0.05). Otherwise it's reported as `(likely noise — p=X, not significant)`.
|
||||
|
||||
With typical variance (cv ≈ 20%), you need:
|
||||
- **n ≥ 5** per build to detect a 35% regression at 95% confidence
|
||||
- **n ≥ 10** per build to detect a 20% regression reliably
|
||||
|
||||
Confidence levels reported: `high` (p < 0.01), `medium` (p < 0.05), `low` (p < 0.1), `none`.
|
||||
|
||||
### Exit codes
|
||||
|
||||
- `0` — all metrics within threshold, or exceeding threshold but not statistically significant
|
||||
- `1` — statistically significant regression detected, or all runs failed
|
||||
|
||||
### Scenarios
|
||||
|
||||
Scenarios are defined in `scripts/chat-simulation/common/perf-scenarios.js` and registered via `registerPerfScenarios()`. There are three categories:
|
||||
|
||||
- **Content-only** — plain streaming responses (e.g. `text-only`, `large-codeblock`, `rapid-stream`)
|
||||
- **Tool-call** — multi-turn scenarios with tool invocations (e.g. `tool-read-file`, `tool-edit-file`)
|
||||
- **Multi-turn user** — multi-turn conversations with user follow-ups, thinking blocks (e.g. `thinking-response`, `multi-turn-user`, `long-conversation`)
|
||||
|
||||
Run `npm run perf:chat -- --help` to see the full list of registered scenario IDs.
|
||||
|
||||
### Metrics collected
|
||||
|
||||
- **Timing:** time to first token, time to complete, time to render complete (includes typewriter animation)
|
||||
- **Rendering:** layout count, layout duration (ms), style recalculation count, forced reflows, long tasks (>50ms), long animation frame count and duration
|
||||
- **Memory:** heap before/after, heap delta post-GC (informational, noisy for single requests)
|
||||
- **Extension host:** heap before/after/delta via CDP inspector
|
||||
|
||||
### Regression triggers vs informational metrics
|
||||
|
||||
Only these metrics trigger a regression failure (when they exceed the threshold with statistical significance):
|
||||
- `timeToFirstToken`, `timeToComplete` — user-perceived latency
|
||||
- `forcedReflowCount` — forced synchronous layouts are always bad
|
||||
- `longTaskCount`, `longAnimationFrameCount` — main thread jank
|
||||
|
||||
These are reported but **informational only** (won't fail CI):
|
||||
- `layoutCount` — inflated by CSS animations; use `layoutDurationMs` instead
|
||||
- `layoutDurationMs` — total layout time from trace (more meaningful than count)
|
||||
- `recalcStyleCount` — inflated by CSS animations (compositor-driven, cheap)
|
||||
- `timeToRenderComplete` — includes typewriter animation tail
|
||||
- Memory/heap metrics — too noisy for single-request benchmarks
|
||||
|
||||
### Statistics
|
||||
|
||||
Results use **IQR-based outlier removal** and **median** (not mean) to handle startup jitter. The **coefficient of variation (cv)** is reported — under 15% is stable, over 15% gets a ⚠ warning. Baseline comparison uses **Welch's t-test** on raw run values to determine statistical significance before flagging regressions. Use 5+ runs to get stable results.
|
||||
|
||||
## Memory leak check
|
||||
|
||||
**Script:** `scripts/chat-simulation/test-chat-mem-leaks.js`
|
||||
**npm:** `npm run perf:chat-leak`
|
||||
|
||||
Launches one VS Code session, sends N messages sequentially, forces GC between each, and measures renderer heap and DOM node count. Uses **linear regression** on the samples to compute per-message growth rate, which is compared against a threshold.
|
||||
|
||||
### Key flags
|
||||
|
||||
| Flag | Default | Description |
|
||||
|---|---|---|
|
||||
| `--messages <n>` / `-n` | `10` | Number of messages to send. More = more accurate slope. |
|
||||
| `--build <path\|ver>` / `-b` | local dev | Build to test. |
|
||||
| `--threshold <MB>` | `2` | Max per-message heap growth in MB. |
|
||||
| `--setting <k=v>` | — | Set a VS Code setting override (repeatable). |
|
||||
| `--verbose` | — | Print per-message heap/DOM counts. |
|
||||
|
||||
### What it measures
|
||||
|
||||
- **Heap growth slope** (MB/message) — linear regression over forced-GC heap samples. A leak shows as sustained positive slope.
|
||||
- **DOM node growth** (nodes/message) — catches rendering leaks where elements aren't cleaned up. Healthy chat virtualizes old messages so node count plateaus.
|
||||
|
||||
### Interpreting results
|
||||
|
||||
- `0.3–1.0 MB/msg` — normal (V8 internal overhead, string interning)
|
||||
- `>2.0 MB/msg` — likely leak, investigate retained objects
|
||||
- DOM nodes stable after first message — normal (chat list virtualization working)
|
||||
- DOM nodes growing linearly — rendering leak, check disposable cleanup
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
scripts/chat-simulation/
|
||||
├── common/
|
||||
│ ├── mock-llm-server.js # Mock CAPI server matching @vscode/copilot-api URL structure
|
||||
│ ├── perf-scenarios.js # Built-in scenario definitions (content, tool-call, multi-turn)
|
||||
│ └── utils.js # Shared: paths, env setup, stats, launch helpers
|
||||
├── config.jsonc # Default config (baseline version, runs, thresholds)
|
||||
├── fixtures/ # TypeScript fixture files used by tool-call scenarios
|
||||
├── test-chat-perf-regression.js
|
||||
└── test-chat-mem-leaks.js
|
||||
```
|
||||
|
||||
### Mock server
|
||||
|
||||
The mock LLM server (`common/mock-llm-server.js`) implements the full CAPI URL structure from `@vscode/copilot-api`'s `DomainService`:
|
||||
|
||||
- `GET /models` — returns model metadata
|
||||
- `POST /models/session` — returns `AutoModeAPIResponse` with `available_models` and `session_token`
|
||||
- `POST /models/session/intent` — model router
|
||||
- `POST /chat/completions` — SSE streaming response matching the scenario
|
||||
- Agent, session, telemetry, and token endpoints
|
||||
|
||||
The copilot extension connects to this server via `IS_SCENARIO_AUTOMATION=1` mode with `overrideCapiUrl` and `overrideProxyUrl` settings. The `vscode-api-tests` extension must be disabled (`--disable-extension=vscode.vscode-api-tests`) because it contributes a duplicate `copilot` vendor that blocks the real extension's language model provider registration.
|
||||
|
||||
### Adding a scenario
|
||||
|
||||
1. Add a new entry to the appropriate object (`CONTENT_SCENARIOS`, `TOOL_CALL_SCENARIOS`, or `MULTI_TURN_SCENARIOS`) in `common/perf-scenarios.js` using the `ScenarioBuilder` API from `common/mock-llm-server.js`
|
||||
2. The scenario is auto-registered by `registerPerfScenarios()` — no manual ID list to update
|
||||
3. Run: `npm run perf:chat -- --scenario your-new-scenario --runs 1 --no-baseline --verbose`
|
||||
|
||||
## Related skills
|
||||
|
||||
- **heap-snapshot-analysis** — When a perf regression or leak check identifies high memory growth, use the heap-snapshot-analysis skill to dig deeper. It can parse `.heapsnapshot` files, compare before/after snapshots, group object deltas, and trace retainer paths to find what keeps disposed objects alive. The chat-perf leak check measures overall heap slope; heap-snapshot-analysis finds the specific objects responsible.
|
||||
- **auto-perf-optimize** — For launching VS Code, driving a scenario, and capturing heap snapshots or CPU profiles automatically before doing low-level analysis.
|
||||
@@ -0,0 +1,467 @@
|
||||
name: Chat Performance Comparison
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
baseline_build:
|
||||
description: "Baseline version or commit SHA (e.g. \"1.116.0\", \"insiders\", \"abc1234\"). Default: config.jsonc baselineBuild."
|
||||
required: false
|
||||
type: string
|
||||
test_build:
|
||||
description: "Branch, PR ref, commit SHA, or version to test (e.g. \"my-feature\", \"refs/pull/12345/head\", \"1.115.0\"). Default: current pipeline branch (probably main)."
|
||||
required: false
|
||||
type: string
|
||||
runs:
|
||||
description: "Runs per scenario"
|
||||
required: false
|
||||
type: number
|
||||
default: 7
|
||||
scenarios:
|
||||
description: "Comma-separated scenario list. Default: all registered scenarios."
|
||||
required: false
|
||||
type: string
|
||||
default: ""
|
||||
threshold:
|
||||
description: "Regression threshold fraction (0.2 = 20%)"
|
||||
required: false
|
||||
type: number
|
||||
default: 0.2
|
||||
skip_leak_check:
|
||||
description: "Skip the memory leak check step"
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
test_settings:
|
||||
description: 'JSON object of VS Code settings for the test build (e.g. {"chat.experimental.smoothStreaming.enabled": true})'
|
||||
required: false
|
||||
type: string
|
||||
default: ""
|
||||
baseline_settings:
|
||||
description: 'JSON object of VS Code settings for the baseline build'
|
||||
required: false
|
||||
type: string
|
||||
default: ""
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: chat-perf-${{ github.run_id }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
# Only set when explicitly provided; otherwise scripts read config.jsonc
|
||||
BASELINE_BUILD_INPUT: ${{ inputs.baseline_build || '' }}
|
||||
TEST_BUILD_INPUT: ${{ inputs.test_build || '' }}
|
||||
PERF_RUNS: ${{ inputs.runs || '' }}
|
||||
PERF_THRESHOLD: ${{ inputs.threshold || '' }}
|
||||
SCENARIOS_INPUT: ${{ inputs.scenarios || '' }}
|
||||
TEST_SETTINGS_INPUT: ${{ inputs.test_settings || '' }}
|
||||
BASELINE_SETTINGS_INPUT: ${{ inputs.baseline_settings || '' }}
|
||||
|
||||
jobs:
|
||||
# ── Shared setup: build once, cache everything ──────────────────────
|
||||
setup:
|
||||
name: Build & Cache
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
outputs:
|
||||
test_is_version: ${{ steps.resolve.outputs.is_version }}
|
||||
test_build_arg: ${{ steps.resolve.outputs.build_arg }}
|
||||
steps:
|
||||
- name: Resolve test build type
|
||||
id: resolve
|
||||
run: |
|
||||
INPUT="$TEST_BUILD_INPUT"
|
||||
if [[ -z "$INPUT" ]]; then
|
||||
echo "is_version=false" >> "$GITHUB_OUTPUT"
|
||||
echo "build_arg=" >> "$GITHUB_OUTPUT"
|
||||
elif [[ "$INPUT" =~ ^[0-9]+\.[0-9]+\.[0-9]+ ]] || [[ "$INPUT" == "insiders" ]] || [[ "$INPUT" == "stable" ]]; then
|
||||
echo "test_build is a version string: $INPUT (will download)"
|
||||
echo "is_version=true" >> "$GITHUB_OUTPUT"
|
||||
echo "build_arg=$INPUT" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "test_build is a git ref: $INPUT (will checkout and build from source)"
|
||||
echo "is_version=false" >> "$GITHUB_OUTPUT"
|
||||
echo "build_arg=" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ steps.resolve.outputs.is_version != 'true' && inputs.test_build || github.ref }}
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version-file: .nvmrc
|
||||
cache: npm
|
||||
|
||||
- name: Install system dependencies
|
||||
run: |
|
||||
sudo apt update -y
|
||||
sudo apt install -y \
|
||||
build-essential pkg-config \
|
||||
libx11-dev libx11-xcb-dev libxkbfile-dev \
|
||||
libnotify-bin libkrb5-dev \
|
||||
xvfb sqlite3 \
|
||||
libnss3 libatk1.0-0 libatk-bridge2.0-0 \
|
||||
libcups2t64 libdrm2 libxcomposite1 libxdamage1 \
|
||||
libxrandr2 libgbm1 libpango-1.0-0 libcairo2 \
|
||||
libasound2t64 libxshmfence1 libgtk-3-0
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Install build dependencies
|
||||
run: npm ci
|
||||
working-directory: build
|
||||
|
||||
- name: Transpile source
|
||||
run: npm run transpile-client
|
||||
|
||||
- name: Build copilot extension
|
||||
run: npm run compile
|
||||
working-directory: extensions/copilot
|
||||
|
||||
- name: Download Electron
|
||||
run: node build/lib/preLaunch.ts
|
||||
|
||||
- name: Cache Electron
|
||||
uses: actions/cache/save@v4
|
||||
with:
|
||||
path: ~/.cache/electron
|
||||
key: electron-${{ runner.os }}-${{ hashFiles('.nvmrc', 'package.json') }}
|
||||
|
||||
- name: Install Playwright Chromium
|
||||
run: npx playwright install chromium
|
||||
|
||||
- name: Cache Playwright
|
||||
uses: actions/cache/save@v4
|
||||
with:
|
||||
path: ~/.cache/ms-playwright
|
||||
key: playwright-${{ runner.os }}-${{ hashFiles('package.json') }}
|
||||
|
||||
- name: Upload build output
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: build-output
|
||||
path: |
|
||||
out/
|
||||
extensions/copilot/dist/
|
||||
retention-days: 1
|
||||
|
||||
# ── Perf comparison (split across matrix groups) ─────────────────────
|
||||
chat-perf:
|
||||
name: Chat Perf (${{ matrix.group }})
|
||||
needs: setup
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 60
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
group: [ 1, 2, 3, 4 ]
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ needs.setup.outputs.test_is_version != 'true' && inputs.test_build || github.ref }}
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version-file: .nvmrc
|
||||
cache: npm
|
||||
|
||||
- name: Install system dependencies
|
||||
run: |
|
||||
sudo apt update -y
|
||||
sudo apt install -y \
|
||||
build-essential pkg-config \
|
||||
libx11-dev libx11-xcb-dev libxkbfile-dev \
|
||||
libnotify-bin libkrb5-dev \
|
||||
xvfb sqlite3 \
|
||||
libnss3 libatk1.0-0 libatk-bridge2.0-0 \
|
||||
libcups2t64 libdrm2 libxcomposite1 libxdamage1 \
|
||||
libxrandr2 libgbm1 libpango-1.0-0 libcairo2 \
|
||||
libasound2t64 libxshmfence1 libgtk-3-0
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Download build output
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
name: build-output
|
||||
|
||||
- name: Restore Electron cache
|
||||
uses: actions/cache/restore@v4
|
||||
with:
|
||||
path: ~/.cache/electron
|
||||
key: electron-${{ runner.os }}-${{ hashFiles('.nvmrc', 'package.json') }}
|
||||
|
||||
- name: Download Electron
|
||||
run: node build/lib/preLaunch.ts
|
||||
|
||||
- name: Restore Playwright cache
|
||||
uses: actions/cache/restore@v4
|
||||
with:
|
||||
path: ~/.cache/ms-playwright
|
||||
key: playwright-${{ runner.os }}-${{ hashFiles('package.json') }}
|
||||
|
||||
- name: Install Playwright Chromium
|
||||
run: npx playwright install chromium
|
||||
|
||||
- name: Resolve scenario group
|
||||
id: scenarios
|
||||
run: |
|
||||
node -e "
|
||||
const fs = require('fs');
|
||||
require('./scripts/chat-simulation/common/perf-scenarios').registerPerfScenarios();
|
||||
const { getScenarioIds } = require('./scripts/chat-simulation/common/mock-llm-server');
|
||||
const userInput = process.env.SCENARIOS_INPUT || '';
|
||||
const allScens = userInput
|
||||
? userInput.split(',').map(s => s.trim()).filter(Boolean)
|
||||
: getScenarioIds();
|
||||
const groups = 4;
|
||||
const group = parseInt(process.env.MATRIX_GROUP, 10);
|
||||
// Distribute scenarios round-robin across groups
|
||||
const groupScens = allScens.filter((_, i) => (i % groups) + 1 === group);
|
||||
if (groupScens.length === 0) {
|
||||
console.log('No scenarios for group ' + group);
|
||||
fs.appendFileSync(process.env.GITHUB_OUTPUT, 'skip=true\n');
|
||||
} else {
|
||||
const args = groupScens.map(s => '--scenario ' + s).join(' ');
|
||||
fs.appendFileSync(process.env.GITHUB_OUTPUT, 'skip=false\n');
|
||||
fs.appendFileSync(process.env.GITHUB_OUTPUT, 'args=' + args + '\n');
|
||||
console.log('Group ' + group + ' (' + groupScens.length + '/' + allScens.length + '): ' + groupScens.join(', '));
|
||||
}
|
||||
"
|
||||
env:
|
||||
MATRIX_GROUP: ${{ matrix.group }}
|
||||
|
||||
- name: Run chat perf comparison
|
||||
id: perf
|
||||
if: steps.scenarios.outputs.skip != 'true'
|
||||
env:
|
||||
SCENARIO_ARGS: ${{ steps.scenarios.outputs.args }}
|
||||
run: |
|
||||
PERF_ARGS=("--ci")
|
||||
if [[ -n "$BASELINE_BUILD_INPUT" ]]; then
|
||||
PERF_ARGS+=("--baseline-build" "$BASELINE_BUILD_INPUT")
|
||||
fi
|
||||
TEST_BUILD_ARG="${{ needs.setup.outputs.test_build_arg }}"
|
||||
if [[ -n "$TEST_BUILD_ARG" ]]; then
|
||||
PERF_ARGS+=("--build" "$TEST_BUILD_ARG")
|
||||
fi
|
||||
if [[ -n "$PERF_RUNS" ]]; then
|
||||
PERF_ARGS+=("--runs" "$PERF_RUNS")
|
||||
fi
|
||||
if [[ -n "$PERF_THRESHOLD" ]]; then
|
||||
PERF_ARGS+=("--threshold" "$PERF_THRESHOLD")
|
||||
fi
|
||||
PERF_ARGS+=("--production-build")
|
||||
|
||||
# Convert JSON settings objects to --test-setting / --baseline-setting flags
|
||||
if [[ -n "$TEST_SETTINGS_INPUT" ]]; then
|
||||
while IFS='=' read -r key value; do
|
||||
PERF_ARGS+=("--test-setting" "$key=$value")
|
||||
done < <(node -e "const s=JSON.parse(process.env.TEST_SETTINGS_INPUT); for (const [k,v] of Object.entries(s)) console.log(k+'='+v)")
|
||||
fi
|
||||
if [[ -n "$BASELINE_SETTINGS_INPUT" ]]; then
|
||||
while IFS='=' read -r key value; do
|
||||
PERF_ARGS+=("--baseline-setting" "$key=$value")
|
||||
done < <(node -e "const s=JSON.parse(process.env.BASELINE_SETTINGS_INPUT); for (const [k,v] of Object.entries(s)) console.log(k+'='+v)")
|
||||
fi
|
||||
|
||||
# Split SCENARIO_ARGS on whitespace into array elements
|
||||
read -ra SCENARIO_ARR <<< "$SCENARIO_ARGS"
|
||||
|
||||
set +eo pipefail
|
||||
xvfb-run node scripts/chat-simulation/test-chat-perf-regression.js \
|
||||
"${PERF_ARGS[@]}" \
|
||||
"${SCENARIO_ARR[@]}" \
|
||||
2>&1 | tee perf-output.log
|
||||
echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Upload perf results
|
||||
if: always() && steps.scenarios.outputs.skip != 'true'
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: perf-results-${{ matrix.group }}
|
||||
include-hidden-files: true
|
||||
path: |
|
||||
perf-output.log
|
||||
.chat-simulation-data/
|
||||
retention-days: 30
|
||||
|
||||
- name: Check for regressions
|
||||
if: always() && steps.perf.outputs.exit_code != '' &&
|
||||
steps.perf.outputs.exit_code != '0'
|
||||
run: |
|
||||
echo "::error::Chat perf regression detected (exit code ${{ steps.perf.outputs.exit_code }}). See perf-output.log for details."
|
||||
exit 1
|
||||
|
||||
# ── Memory leak check (runs in parallel with perf) ──────────────────
|
||||
leak-check:
|
||||
name: Leak Check
|
||||
needs: setup
|
||||
if: inputs.skip_leak_check != true
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 60
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ needs.setup.outputs.test_is_version != 'true' && inputs.test_build || github.ref }}
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version-file: .nvmrc
|
||||
cache: npm
|
||||
|
||||
- name: Install system dependencies
|
||||
run: |
|
||||
sudo apt update -y
|
||||
sudo apt install -y \
|
||||
build-essential pkg-config \
|
||||
libx11-dev libx11-xcb-dev libxkbfile-dev \
|
||||
libnotify-bin libkrb5-dev \
|
||||
xvfb \
|
||||
libnss3 libatk1.0-0 libatk-bridge2.0-0 \
|
||||
libcups2t64 libdrm2 libxcomposite1 libxdamage1 \
|
||||
libxrandr2 libgbm1 libpango-1.0-0 libcairo2 \
|
||||
libasound2t64 libxshmfence1 libgtk-3-0
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Download build output
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
name: build-output
|
||||
|
||||
- name: Restore Electron cache
|
||||
uses: actions/cache/restore@v4
|
||||
with:
|
||||
path: ~/.cache/electron
|
||||
key: electron-${{ runner.os }}-${{ hashFiles('.nvmrc', 'package.json') }}
|
||||
|
||||
- name: Download Electron
|
||||
run: node build/lib/preLaunch.ts
|
||||
|
||||
- name: Restore Playwright cache
|
||||
uses: actions/cache/restore@v4
|
||||
with:
|
||||
path: ~/.cache/ms-playwright
|
||||
key: playwright-${{ runner.os }}-${{ hashFiles('package.json') }}
|
||||
|
||||
- name: Install Playwright Chromium
|
||||
run: npx playwright install chromium
|
||||
|
||||
- name: Run memory leak check
|
||||
id: leak
|
||||
run: |
|
||||
LEAK_ARGS="--verbose --ci"
|
||||
TEST_BUILD_ARG="${{ needs.setup.outputs.test_build_arg }}"
|
||||
if [[ -n "$TEST_BUILD_ARG" ]]; then
|
||||
LEAK_ARGS="$LEAK_ARGS --build $TEST_BUILD_ARG"
|
||||
fi
|
||||
|
||||
set +eo pipefail
|
||||
xvfb-run node scripts/chat-simulation/test-chat-mem-leaks.js \
|
||||
$LEAK_ARGS \
|
||||
2>&1 | tee leak-output.log
|
||||
echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Upload leak results
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: leak-results
|
||||
include-hidden-files: true
|
||||
path: |
|
||||
leak-output.log
|
||||
.chat-simulation-data/chat-simulation-leak-results.json
|
||||
.chat-simulation-data/ci-summary-leak.md
|
||||
retention-days: 30
|
||||
|
||||
- name: Check for leaks
|
||||
if: always() && steps.leak.outputs.exit_code != '' &&
|
||||
steps.leak.outputs.exit_code != '0'
|
||||
run: |
|
||||
echo "::error::Chat memory leak detected (exit code ${{ steps.leak.outputs.exit_code }}). See leak-output.log for details."
|
||||
exit 1
|
||||
|
||||
# ── Report: collect results, write summary, fail on regression ──────
|
||||
report:
|
||||
name: Report
|
||||
needs: [ chat-perf, leak-check ]
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ needs.setup.outputs.test_is_version != 'true' && inputs.test_build || github.ref }}
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version-file: .nvmrc
|
||||
|
||||
- name: Download all perf results
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
pattern: perf-results-*
|
||||
path: perf-results
|
||||
|
||||
- name: Download leak results
|
||||
if: inputs.skip_leak_check != true && needs.leak-check.result != 'skipped'
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
name: leak-results
|
||||
path: leak-results
|
||||
continue-on-error: true
|
||||
|
||||
- name: Generate unified summary
|
||||
run: |
|
||||
LEAK_ARG=""
|
||||
if [[ -f leak-results/.chat-simulation-data/ci-summary-leak.md ]]; then
|
||||
LEAK_ARG="--leak-summary leak-results/.chat-simulation-data/ci-summary-leak.md"
|
||||
fi
|
||||
|
||||
node scripts/chat-simulation/merge-ci-summary.js \
|
||||
--results-dir perf-results \
|
||||
--output ci-summary.md \
|
||||
$LEAK_ARG
|
||||
|
||||
cat ci-summary.md >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
- name: Upload CI summary
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: chat-perf-summary
|
||||
path: ci-summary.md
|
||||
retention-days: 30
|
||||
|
||||
- name: Fail on regression
|
||||
id: regression
|
||||
if: needs.chat-perf.result == 'failure' || (inputs.skip_leak_check != true &&
|
||||
needs.leak-check.result == 'failure')
|
||||
run: |
|
||||
if [[ "${{ needs.chat-perf.result }}" == "failure" ]]; then
|
||||
echo "::error::Chat performance regression detected. See job summary for details."
|
||||
fi
|
||||
if [[ "${{ inputs.skip_leak_check }}" != "true" && "${{ needs.leak-check.result }}" == "failure" ]]; then
|
||||
echo "::error::Chat memory leak detected. See leak-output.log for details."
|
||||
fi
|
||||
exit 1
|
||||
@@ -25,6 +25,7 @@ product.overrides.json
|
||||
*.snap.actual
|
||||
*.tsbuildinfo
|
||||
.vscode-test
|
||||
.chat-simulation-data
|
||||
vscode-telemetry-docs/
|
||||
test-output.json
|
||||
test/componentFixtures/.screenshots/*
|
||||
|
||||
@@ -163,6 +163,7 @@ export const copyrightFilter = Object.freeze<string[]>([
|
||||
'**',
|
||||
'!**/*.desktop',
|
||||
'!**/*.json',
|
||||
'!**/*.jsonc',
|
||||
'!**/*.jsonl',
|
||||
'!**/*.html',
|
||||
'!**/*.template',
|
||||
|
||||
@@ -79,6 +79,8 @@
|
||||
"extensions-ci": "npm run gulp extensions-ci",
|
||||
"extensions-ci-pr": "npm run gulp extensions-ci-pr",
|
||||
"perf": "node scripts/code-perf.js",
|
||||
"perf:chat": "node scripts/chat-simulation/test-chat-perf-regression.js",
|
||||
"perf:chat-leak": "node scripts/chat-simulation/test-chat-mem-leaks.js",
|
||||
"copilot:setup": "npm --prefix extensions/copilot run setup",
|
||||
"copilot:get_token": "npm --prefix extensions/copilot run get_token",
|
||||
"update-build-ts-version": "npm install -D typescript@next && npm install -D @typescript/native-preview && (cd build && npm run typecheck)",
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,800 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
// @ts-check
|
||||
|
||||
/**
|
||||
* Built-in scenario definitions for chat performance benchmarks and leak checks.
|
||||
*
|
||||
* Each test file imports this module and calls `registerScenario()` for the
|
||||
* scenarios it needs, keeping scenario ownership close to the test that uses it.
|
||||
*/
|
||||
|
||||
const path = require('path');
|
||||
const { ScenarioBuilder, registerScenario } = require('./mock-llm-server');
|
||||
|
||||
const FIXTURES_DIR = path.join(__dirname, '..', 'fixtures');
|
||||
|
||||
/**
|
||||
* @typedef {{
|
||||
* description: string,
|
||||
* chunks: import('./mock-llm-server').StreamChunk[],
|
||||
* }} ContentScenarioDef
|
||||
*
|
||||
* @typedef {{
|
||||
* description: string,
|
||||
* scenario: import('./mock-llm-server').MultiTurnScenario,
|
||||
* }} MultiTurnScenarioDef
|
||||
*/
|
||||
|
||||
// -- Content-only scenarios ---------------------------------------------------
|
||||
|
||||
/** @type {Record<string, ContentScenarioDef>} */
|
||||
const CONTENT_SCENARIOS = {
|
||||
'text-only': {
|
||||
description: 'Plain text, 4 paragraphs',
|
||||
chunks: new ScenarioBuilder()
|
||||
.stream([
|
||||
'Here is an explanation of the code you selected:\n\n',
|
||||
'The function `processItems` iterates over the input array and applies a transformation to each element. ',
|
||||
'It uses a `Map` to track previously seen values, which allows it to deduplicate results efficiently in O(n) time.\n\n',
|
||||
'The algorithm works in a single pass: for every element, it computes the transformed value, ',
|
||||
'checks membership in the set, and conditionally appends to the output array. ',
|
||||
'This is a common pattern in data processing pipelines where uniqueness constraints must be maintained.\n\n',
|
||||
'Edge cases to consider include empty arrays, duplicate transformations that produce the same key, ',
|
||||
'and items where the transform function itself is expensive.\n\n',
|
||||
'The time complexity is **O(n)** and the space complexity is **O(n)** in the worst case when all items are unique.\n',
|
||||
], 20)
|
||||
.build(),
|
||||
},
|
||||
|
||||
'large-codeblock': {
|
||||
description: 'Single large TypeScript code block',
|
||||
chunks: new ScenarioBuilder()
|
||||
.stream([
|
||||
'Here is the refactored implementation:\n\n',
|
||||
'```typescript\n',
|
||||
'import { EventEmitter } from "events";\n\n',
|
||||
'interface CacheEntry<T> {\n value: T;\n expiresAt: number;\n accessCount: number;\n}\n\n',
|
||||
'export class LRUCache<K, V> {\n',
|
||||
' private readonly _map = new Map<K, CacheEntry<V>>();\n',
|
||||
' private readonly _emitter = new EventEmitter();\n\n',
|
||||
' constructor(\n private readonly _maxSize: number,\n private readonly _ttlMs: number = 60_000,\n ) {}\n\n',
|
||||
' get(key: K): V | undefined {\n const entry = this._map.get(key);\n if (!entry) { return undefined; }\n',
|
||||
' if (Date.now() > entry.expiresAt) {\n this._map.delete(key);\n this._emitter.emit("evict", key);\n return undefined;\n }\n',
|
||||
' entry.accessCount++;\n this._map.delete(key);\n this._map.set(key, entry);\n return entry.value;\n }\n\n',
|
||||
' set(key: K, value: V): void {\n if (this._map.size >= this._maxSize) {\n',
|
||||
' const oldest = this._map.keys().next().value;\n if (oldest !== undefined) {\n this._map.delete(oldest);\n this._emitter.emit("evict", oldest);\n }\n }\n',
|
||||
' this._map.set(key, { value, expiresAt: Date.now() + this._ttlMs, accessCount: 0 });\n }\n\n',
|
||||
' clear(): void { this._map.clear(); this._emitter.emit("clear"); }\n',
|
||||
' get size(): number { return this._map.size; }\n',
|
||||
' onEvict(listener: (key: K) => void): void { this._emitter.on("evict", listener); }\n}\n',
|
||||
'```\n\n',
|
||||
'The key changes:\n- Added TTL-based expiry with configurable timeout\n- LRU eviction uses Map insertion order\n- EventEmitter notifies on evictions for cache observability\n',
|
||||
], 20)
|
||||
.build(),
|
||||
},
|
||||
|
||||
'many-small-chunks': {
|
||||
description: '200 word-level chunks at 5ms',
|
||||
chunks: (() => {
|
||||
const words = ['Generating detailed analysis:\n\n'];
|
||||
for (let i = 0; i < 200; i++) { words.push(`Word${i} `); }
|
||||
words.push('\n\nAnalysis complete.\n');
|
||||
const b = new ScenarioBuilder();
|
||||
b.stream(words, 5);
|
||||
return b.build();
|
||||
})(),
|
||||
},
|
||||
|
||||
'mixed-content': {
|
||||
description: 'Markdown + code block + fix suggestion',
|
||||
chunks: new ScenarioBuilder()
|
||||
.stream([
|
||||
'## Issue Found\n\n',
|
||||
'The `DisposableStore` is not being disposed in the `deactivate` path, ',
|
||||
'which can lead to memory leaks.\n\n',
|
||||
'### Current Code\n\n',
|
||||
'```typescript\nclass MyService {\n private store = new DisposableStore();\n // missing dispose!\n}\n```\n\n',
|
||||
'### Suggested Fix\n\n',
|
||||
'```typescript\nclass MyService extends Disposable {\n',
|
||||
' private readonly store = this._register(new DisposableStore());\n\n',
|
||||
' override dispose(): void {\n this.store.dispose();\n super.dispose();\n }\n}\n```\n\n',
|
||||
'This ensures the store is cleaned up when the service is disposed via the workbench lifecycle.\n',
|
||||
], 20)
|
||||
.build(),
|
||||
},
|
||||
|
||||
// -- Stress-test scenarios --------------------------------------------
|
||||
|
||||
'many-codeblocks': {
|
||||
description: '10 code blocks, 60 lines each',
|
||||
chunks: (() => {
|
||||
const b = new ScenarioBuilder();
|
||||
b.emit('Here are the implementations for each module:\n\n');
|
||||
for (let i = 0; i < 10; i++) {
|
||||
b.wait(10, `### Module ${i + 1}: \`handler${i}.ts\`\n\n`);
|
||||
b.emit('```typescript\n');
|
||||
const lines = [];
|
||||
for (let j = 0; j < 15; j++) {
|
||||
lines.push(`export function handle${i}_${j}(input: string): string {\n`);
|
||||
lines.push(` const result = input.trim().split('').reverse().join('');\n`);
|
||||
lines.push(` return \`[\${result}] processed by handler ${i}_${j}\`;\n`);
|
||||
lines.push('}\n\n');
|
||||
}
|
||||
b.stream(lines, 5);
|
||||
b.emit('```\n\n');
|
||||
}
|
||||
b.emit('All modules implement the same pattern with unique handler IDs.\n');
|
||||
return b.build();
|
||||
})(),
|
||||
},
|
||||
|
||||
'long-prose': {
|
||||
description: '15 sections, ~3000 words of prose',
|
||||
chunks: (() => {
|
||||
const sentences = [
|
||||
'The architecture follows a layered dependency injection pattern where each service declares its dependencies through constructor parameters. ',
|
||||
'This approach ensures that circular dependencies are detected at compile time rather than at runtime, which significantly reduces debugging overhead. ',
|
||||
'When a service is instantiated, the instantiation service resolves all of its dependencies recursively, creating a directed acyclic graph of service instances. ',
|
||||
'Each service is a singleton within its scope, meaning that multiple consumers of the same service interface receive the same instance. ',
|
||||
'The workbench lifecycle manages the creation and disposal of these services through well-defined phases: creation, restoration, and eventual shutdown. ',
|
||||
'During the restoration phase, services that persist state across sessions reload their data from storage, which may involve asynchronous operations. ',
|
||||
'Contributors register their functionality through extension points, which are processed during the appropriate lifecycle phase. ',
|
||||
'This contribution model allows features to be added without modifying the core workbench code, maintaining a clean separation of concerns. ',
|
||||
];
|
||||
const b = new ScenarioBuilder();
|
||||
b.emit('# Detailed Architecture Analysis\n\n');
|
||||
for (let para = 0; para < 15; para++) {
|
||||
b.wait(15, `## Section ${para + 1}: ${['Overview', 'Design Patterns', 'Service Layer', 'Event System', 'State Management', 'Error Handling', 'Performance', 'Testing', 'Deployment', 'Monitoring', 'Security', 'Extensibility', 'Compatibility', 'Migration', 'Future Work'][para]}\n\n`);
|
||||
const paraSentences = [];
|
||||
for (let s = 0; s < 25; s++) { paraSentences.push(sentences[s % sentences.length]); }
|
||||
b.stream(paraSentences, 8);
|
||||
b.emit('\n\n');
|
||||
}
|
||||
return b.build();
|
||||
})(),
|
||||
},
|
||||
|
||||
'rich-markdown': {
|
||||
description: '6 sections × 5 items, bold/links/code spans',
|
||||
chunks: (() => {
|
||||
const b = new ScenarioBuilder();
|
||||
b.emit('# Comprehensive Code Review Report\n\n');
|
||||
b.wait(15, '> **Summary**: Found 12 issues across 4 severity levels.\n\n');
|
||||
for (let section = 0; section < 6; section++) {
|
||||
b.wait(10, `## ${section + 1}. ${['Critical Issues', 'Performance Concerns', 'Code Style', 'Documentation Gaps', 'Test Coverage', 'Security Review'][section]}\n\n`);
|
||||
for (let item = 0; item < 5; item++) {
|
||||
b.stream([
|
||||
`${item + 1}. **Issue ${section * 5 + item + 1}**: \`${['useState', 'useEffect', 'useMemo', 'useCallback', 'useRef'][item]}\` in \`src/components/Widget${item}.tsx\`\n`,
|
||||
` - Severity: ${['[Critical]', '[Warning]', '[Info]', '[Suggestion]', '[Note]'][item]}\n`,
|
||||
` - The current implementation uses *unnecessary re-renders* due to missing dependency arrays.\n`,
|
||||
` - See [React docs](https://react.dev/reference) and the [\`useMemo\` guide](https://react.dev/reference/react/useMemo).\n`,
|
||||
` - Fix: wrap in \`useCallback\` or extract to a ***separate memoized component***.\n\n`,
|
||||
], 10);
|
||||
}
|
||||
b.emit('---\n\n');
|
||||
}
|
||||
b.emit('> *Report generated automatically. Please review all suggestions before applying.*\n');
|
||||
return b.build();
|
||||
})(),
|
||||
},
|
||||
|
||||
'giant-codeblock': {
|
||||
description: '40 classes in one fenced code block',
|
||||
chunks: (() => {
|
||||
const b = new ScenarioBuilder();
|
||||
b.emit('Here is the complete implementation:\n\n```typescript\n');
|
||||
b.stream([
|
||||
'import { Disposable, DisposableStore } from "vs/base/common/lifecycle";\n',
|
||||
'import { Emitter, Event } from "vs/base/common/event";\n',
|
||||
'import { URI } from "vs/base/common/uri";\n\n',
|
||||
], 10);
|
||||
for (let i = 0; i < 40; i++) {
|
||||
b.stream([
|
||||
`export class Service${i} extends Disposable {\n`,
|
||||
` private readonly _onDidChange = this._register(new Emitter<void>());\n`,
|
||||
` readonly onDidChange: Event<void> = this._onDidChange.event;\n\n`,
|
||||
` private _value: string = '';\n`,
|
||||
` get value(): string { return this._value; }\n\n`,
|
||||
` async update(uri: URI): Promise<void> {\n`,
|
||||
` this._value = uri.toString();\n`,
|
||||
` this._onDidChange.fire();\n`,
|
||||
` }\n`,
|
||||
'}\n\n',
|
||||
], 5);
|
||||
}
|
||||
b.emit('```\n\nThis defines 40 service classes following the standard VS Code pattern.\n');
|
||||
return b.build();
|
||||
})(),
|
||||
},
|
||||
|
||||
'rapid-stream': {
|
||||
description: '1000 tokens at 2ms (streaming stress test)',
|
||||
chunks: (() => {
|
||||
const b = new ScenarioBuilder();
|
||||
const words = [];
|
||||
for (let i = 0; i < 1000; i++) { words.push(`w${i} `); }
|
||||
// Very fast inter-chunk delay to stress the streaming pipeline
|
||||
b.stream(words, 2);
|
||||
return b.build();
|
||||
})(),
|
||||
},
|
||||
|
||||
'file-links': {
|
||||
description: '32 file references with line links',
|
||||
chunks: (() => {
|
||||
const files = [
|
||||
'src/vs/workbench/contrib/chat/browser/chatListRenderer.ts',
|
||||
'src/vs/workbench/contrib/chat/common/chatService/chatServiceImpl.ts',
|
||||
'src/vs/workbench/contrib/chat/browser/widget/input/chatInputPart.ts',
|
||||
'src/vs/workbench/contrib/chat/common/chatPerf.ts',
|
||||
'src/vs/base/common/lifecycle.ts',
|
||||
'src/vs/base/common/event.ts',
|
||||
'src/vs/platform/instantiation/common/instantiation.ts',
|
||||
'src/vs/workbench/services/extensions/common/abstractExtensionService.ts',
|
||||
'src/vs/workbench/api/common/extHostLanguageModels.ts',
|
||||
'src/vs/workbench/contrib/chat/common/languageModels.ts',
|
||||
'src/vs/editor/browser/widget/codeEditor/editor.ts',
|
||||
'src/vs/workbench/browser/parts/editor/editorGroupView.ts',
|
||||
];
|
||||
const b = new ScenarioBuilder();
|
||||
b.emit('I found references to the disposable pattern across the following files:\n\n');
|
||||
for (let i = 0; i < files.length; i++) {
|
||||
const line = Math.floor(Math.random() * 500) + 1;
|
||||
b.stream([
|
||||
`${i + 1}. [${files[i]}](${files[i]}#L${line}) -- `,
|
||||
`Line ${line}: uses \`DisposableStore\` with ${Math.floor(Math.random() * 10) + 1} registrations\n`,
|
||||
], 15);
|
||||
}
|
||||
b.wait(10, '\nAdditionally, the following files import from `vs/base/common/lifecycle`:\n\n');
|
||||
for (let i = 0; i < 20; i++) {
|
||||
const depth = ['base', 'platform', 'editor', 'workbench'][i % 4];
|
||||
const area = ['common', 'browser', 'node', 'electron-browser'][i % 4];
|
||||
const name = ['service', 'provider', 'contribution', 'handler', 'manager'][i % 5];
|
||||
const file = `src/vs/${depth}/${area}/${name}${i}.ts`;
|
||||
b.stream([
|
||||
`- [${file}](${file}#L${i * 10 + 5})`,
|
||||
` -- imports \`Disposable\`, \`DisposableStore\`\n`,
|
||||
], 12);
|
||||
}
|
||||
b.emit('\nTotal: 32 files reference the disposable pattern.\n');
|
||||
return b.build();
|
||||
})(),
|
||||
},
|
||||
};
|
||||
|
||||
// -- Tool call scenarios ------------------------------------------------------
|
||||
|
||||
/** @type {Record<string, MultiTurnScenarioDef>} */
|
||||
const TOOL_CALL_SCENARIOS = {
|
||||
'tool-read-file': {
|
||||
description: 'Read 8 files across 2 tool-call rounds',
|
||||
scenario: /** @type {import('./mock-llm-server').MultiTurnScenario} */ ((() => {
|
||||
const filesToRead = [
|
||||
'_chatperf_lifecycle.ts',
|
||||
'_chatperf_event.ts',
|
||||
'_chatperf_uri.ts',
|
||||
'_chatperf_errors.ts',
|
||||
'_chatperf_async.ts',
|
||||
'_chatperf_strings.ts',
|
||||
'_chatperf_arrays.ts',
|
||||
'_chatperf_types.ts',
|
||||
];
|
||||
// Round 1: parallel read of first 4 files
|
||||
// Round 2: parallel read of next 4 files
|
||||
// Round 3: final content response
|
||||
return {
|
||||
type: 'multi-turn',
|
||||
turns: [
|
||||
{
|
||||
kind: 'tool-calls',
|
||||
toolCalls: filesToRead.slice(0, 4).map(f => ({
|
||||
toolNamePattern: /read.?file/i,
|
||||
arguments: { filePath: path.join(FIXTURES_DIR, f), startLine: 1, endLine: 50 },
|
||||
})),
|
||||
},
|
||||
{
|
||||
kind: 'tool-calls',
|
||||
toolCalls: filesToRead.slice(4).map(f => ({
|
||||
toolNamePattern: /read.?file/i,
|
||||
arguments: { filePath: path.join(FIXTURES_DIR, f), startLine: 1, endLine: 50 },
|
||||
})),
|
||||
},
|
||||
{
|
||||
kind: 'content',
|
||||
chunks: new ScenarioBuilder()
|
||||
.wait(20, '## Analysis of VS Code Base Utilities\n\n')
|
||||
.stream([
|
||||
'I read 8 core utility files from `src/vs/base/common/`. Here is a summary:\n\n',
|
||||
'### lifecycle.ts\n',
|
||||
'The `Disposable` base class provides the standard lifecycle pattern. Components register cleanup ',
|
||||
'handlers via `this._register()` which are automatically disposed when the parent is disposed.\n\n',
|
||||
'### event.ts\n',
|
||||
'The `Emitter` class implements the observer pattern. `Event.once()`, `Event.map()`, and `Event.filter()` ',
|
||||
'provide functional combinators for composing event streams.\n\n',
|
||||
'### uri.ts\n',
|
||||
'`URI` is an immutable representation of a resource identifier with scheme, authority, path, query, and fragment.\n\n',
|
||||
'### errors.ts\n',
|
||||
'Central error handling with `onUnexpectedError()` and `isCancellationError()` for distinguishing user cancellation.\n\n',
|
||||
'### async.ts\n',
|
||||
'`Throttler`, `Delayer`, `RunOnceScheduler`, and `Queue` manage async operation scheduling and deduplication.\n\n',
|
||||
'### strings.ts\n',
|
||||
'String utilities including `format()`, `escape()`, `startsWith()`, and `endsWith()` for common string operations.\n\n',
|
||||
'### arrays.ts\n',
|
||||
'Array helpers like `coalesce()`, `groupBy()`, `distinct()`, and binary search implementations.\n\n',
|
||||
'### types.ts\n',
|
||||
'Type guards and assertion helpers: `isString()`, `isNumber()`, `assertType()`, `assertIsDefined()`.\n',
|
||||
], 15)
|
||||
.build(),
|
||||
},
|
||||
],
|
||||
};
|
||||
})()),
|
||||
},
|
||||
|
||||
'tool-edit-file': {
|
||||
description: 'Read 3 files, edit 2 (read + write rounds)',
|
||||
scenario: /** @type {import('./mock-llm-server').MultiTurnScenario} */ ((() => {
|
||||
const readFiles = [
|
||||
'_chatperf_lifecycle.ts',
|
||||
'_chatperf_event.ts',
|
||||
'_chatperf_errors.ts',
|
||||
];
|
||||
return {
|
||||
type: 'multi-turn',
|
||||
turns: [
|
||||
// Round 1: read all 3 files in parallel
|
||||
{
|
||||
kind: 'tool-calls',
|
||||
toolCalls: readFiles.map(f => ({
|
||||
toolNamePattern: /read.?file/i,
|
||||
arguments: { filePath: path.join(FIXTURES_DIR, f), startLine: 1, endLine: 40 },
|
||||
})),
|
||||
},
|
||||
// Round 2: edit 2 files in parallel
|
||||
{
|
||||
kind: 'tool-calls',
|
||||
toolCalls: [
|
||||
{
|
||||
toolNamePattern: /insert.?edit|replace.?string|apply.?patch/i,
|
||||
arguments: {
|
||||
filePath: path.join(FIXTURES_DIR, '_chatperf_lifecycle.ts'),
|
||||
explanation: 'Update the benchmark marker comment in lifecycle.ts',
|
||||
code: '// perf-benchmark-marker (updated)',
|
||||
},
|
||||
},
|
||||
{
|
||||
toolNamePattern: /insert.?edit|replace.?string|apply.?patch/i,
|
||||
arguments: {
|
||||
filePath: path.join(FIXTURES_DIR, '_chatperf_event.ts'),
|
||||
explanation: 'Update the benchmark marker comment in event.ts',
|
||||
code: '// perf-benchmark-marker (updated)',
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
// Round 3: final content
|
||||
{
|
||||
kind: 'content',
|
||||
chunks: new ScenarioBuilder()
|
||||
.wait(20, '## Edits Applied\n\n')
|
||||
.stream([
|
||||
'I read 3 files and applied edits to 2 of them:\n\n',
|
||||
'### Files read:\n',
|
||||
'1. `src/vs/base/common/lifecycle.ts` — Disposable pattern and lifecycle management\n',
|
||||
'2. `src/vs/base/common/event.ts` — Event emitter and observer pattern\n',
|
||||
'3. `src/vs/base/common/errors.ts` — Error handling utilities\n\n',
|
||||
'### Edits applied:\n',
|
||||
'1. **lifecycle.ts** — Updated the benchmark marker comment\n',
|
||||
'2. **event.ts** — Updated the benchmark marker comment\n\n',
|
||||
'Both files follow the standard VS Code pattern of using `Disposable` as a base class ',
|
||||
'with `_register()` for lifecycle management. The edits were minimal and localized.\n',
|
||||
], 20)
|
||||
.build(),
|
||||
},
|
||||
],
|
||||
};
|
||||
})()),
|
||||
},
|
||||
|
||||
'tool-terminal': {
|
||||
description: 'Run commands, read output, fix + rerun',
|
||||
scenario: /** @type {import('./mock-llm-server').MultiTurnScenario} */ ({
|
||||
type: 'multi-turn',
|
||||
turns: [
|
||||
// Round 1: run initial commands (install + build)
|
||||
{
|
||||
kind: 'tool-calls',
|
||||
toolCalls: [
|
||||
{
|
||||
toolNamePattern: /run.?in.?terminal|execute.?command/i,
|
||||
arguments: {
|
||||
command: 'echo "Installing dependencies..." && echo "added 1631 packages in 6m"',
|
||||
explanation: 'Install project dependencies',
|
||||
goal: 'Install dependencies',
|
||||
mode: 'sync',
|
||||
timeout: 30000,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
// Round 2: run test command
|
||||
{
|
||||
kind: 'tool-calls',
|
||||
toolCalls: [
|
||||
{
|
||||
toolNamePattern: /run.?in.?terminal|execute.?command/i,
|
||||
arguments: {
|
||||
command: 'echo "Running unit tests..." && echo " 42 passing (3s)" && echo " 2 failing" && echo "" && echo " 1) ChatService should dispose listeners" && echo " AssertionError: expected 0 to equal 1" && echo " 2) ChatModel should clear on new session" && echo " TypeError: Cannot read property dispose of undefined"',
|
||||
explanation: 'Run the unit test suite to check for failures',
|
||||
goal: 'Run tests',
|
||||
mode: 'sync',
|
||||
timeout: 60000,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
// Round 3: read the failing test file for context
|
||||
{
|
||||
kind: 'tool-calls',
|
||||
toolCalls: [
|
||||
{
|
||||
toolNamePattern: /read.?file/i,
|
||||
arguments: { filePath: path.join(FIXTURES_DIR, '_chatperf_lifecycle.ts'), startLine: 1, endLine: 50 },
|
||||
},
|
||||
],
|
||||
},
|
||||
// Round 4: fix the issue with an edit
|
||||
{
|
||||
kind: 'tool-calls',
|
||||
toolCalls: [
|
||||
{
|
||||
toolNamePattern: /insert.?edit|replace.?string|apply.?patch/i,
|
||||
arguments: {
|
||||
filePath: path.join(FIXTURES_DIR, '_chatperf_lifecycle.ts'),
|
||||
explanation: 'Fix the dispose call in the test',
|
||||
code: '// perf-benchmark-marker (fixed)',
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
// Round 5: re-run tests to confirm
|
||||
{
|
||||
kind: 'tool-calls',
|
||||
toolCalls: [
|
||||
{
|
||||
toolNamePattern: /run.?in.?terminal|execute.?command/i,
|
||||
arguments: {
|
||||
command: 'echo "Running unit tests..." && echo " 44 passing (3s)" && echo " 0 failing"',
|
||||
explanation: 'Re-run tests to verify the fix',
|
||||
goal: 'Verify fix',
|
||||
mode: 'sync',
|
||||
timeout: 60000,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
// Round 6: final summary
|
||||
{
|
||||
kind: 'content',
|
||||
chunks: new ScenarioBuilder()
|
||||
.wait(20, '## Test Failures Fixed\n\n')
|
||||
.stream([
|
||||
'I found and fixed 2 test failures:\n\n',
|
||||
'### Root Cause\n',
|
||||
'The `ChatService` was not properly disposing event listeners when a session was cleared. ',
|
||||
'The `dispose()` method was missing a call to `this._store.dispose()`.\n\n',
|
||||
'### Changes Made\n',
|
||||
'Updated `lifecycle.ts` to properly chain disposal:\n\n',
|
||||
'```typescript\n',
|
||||
'override dispose(): void {\n',
|
||||
' this._store.dispose();\n',
|
||||
' super.dispose();\n',
|
||||
'}\n',
|
||||
'```\n\n',
|
||||
'### Test Results\n',
|
||||
'- **Before**: 42 passing, 2 failing\n',
|
||||
'- **After**: 44 passing, 0 failing\n\n',
|
||||
'All tests pass now. The fix ensures listeners are cleaned up during session transitions.\n',
|
||||
], 15)
|
||||
.build(),
|
||||
},
|
||||
],
|
||||
}),
|
||||
},
|
||||
};
|
||||
|
||||
// -- Multi-turn user conversation scenarios -----------------------------------
|
||||
|
||||
/** @type {Record<string, MultiTurnScenarioDef>} */
|
||||
const MULTI_TURN_SCENARIOS = {
|
||||
'thinking-response': {
|
||||
description: 'Thinking block before content response',
|
||||
scenario: /** @type {import('./mock-llm-server').MultiTurnScenario} */ ({
|
||||
type: 'multi-turn',
|
||||
turns: [
|
||||
{
|
||||
kind: 'thinking',
|
||||
thinkingChunks: new ScenarioBuilder()
|
||||
.stream([
|
||||
'Let me analyze this code carefully. ',
|
||||
'The user is asking about the lifecycle pattern in VS Code. ',
|
||||
'I should look at the Disposable base class and how it manages cleanup. ',
|
||||
'The key methods are _register(), dispose(), and the DisposableStore pattern. ',
|
||||
'I need to read the file first to give an accurate explanation.',
|
||||
], 15)
|
||||
.build(),
|
||||
chunks: new ScenarioBuilder()
|
||||
.wait(20, 'I\'ll start by reading the file to understand its structure.\n\n')
|
||||
.stream([
|
||||
'The `Disposable` base class in `lifecycle.ts` provides a standard pattern ',
|
||||
'for managing resources. It uses a `DisposableStore` internally to track ',
|
||||
'all registered disposables and clean them up on `dispose()`.\n',
|
||||
], 20)
|
||||
.build(),
|
||||
},
|
||||
],
|
||||
}),
|
||||
},
|
||||
|
||||
'multi-turn-user': {
|
||||
description: '2 user follow-ups with thinking + code',
|
||||
scenario: /** @type {import('./mock-llm-server').MultiTurnScenario} */ ({
|
||||
type: 'multi-turn',
|
||||
turns: [
|
||||
// Turn 1: Model reads a file
|
||||
{
|
||||
kind: 'tool-calls',
|
||||
toolCalls: [
|
||||
{
|
||||
toolNamePattern: /read.?file/i,
|
||||
arguments: {
|
||||
filePath: path.join(FIXTURES_DIR, '_chatperf_lifecycle.ts'),
|
||||
offset: 1,
|
||||
limit: 50,
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
// Turn 2: Model responds with analysis
|
||||
{
|
||||
kind: 'content',
|
||||
chunks: new ScenarioBuilder()
|
||||
.wait(20, 'I\'ve read the file. Here\'s what I found:\n\n')
|
||||
.stream([
|
||||
'The `Disposable` class is the base for lifecycle management. ',
|
||||
'It internally holds a `DisposableStore` via `this._store`. ',
|
||||
'Subclasses call `this._register()` to track their own disposables.\n\n',
|
||||
'Would you like me to explain any specific part in more detail?\n',
|
||||
], 20)
|
||||
.build(),
|
||||
},
|
||||
// Turn 3: User follow-up (injected by test harness, not served by mock)
|
||||
{
|
||||
kind: 'user',
|
||||
message: 'Yes, explain the MutableDisposable pattern',
|
||||
},
|
||||
// Turn 4: Model responds with thinking, then content
|
||||
{
|
||||
kind: 'thinking',
|
||||
thinkingChunks: new ScenarioBuilder()
|
||||
.stream([
|
||||
'The user wants to understand MutableDisposable specifically. ',
|
||||
'Let me recall the key aspects: it holds a single disposable that can be swapped. ',
|
||||
'When a new value is set, the old one is automatically disposed. ',
|
||||
'This is useful for things like event listener subscriptions that need to be replaced.',
|
||||
], 10)
|
||||
.build(),
|
||||
chunks: new ScenarioBuilder()
|
||||
.wait(15, '## MutableDisposable\n\n')
|
||||
.stream([
|
||||
'`MutableDisposable<T>` holds a **single disposable** that can be swapped at any time. ',
|
||||
'When you set a new value via `.value = newDisposable`, the previous value is automatically disposed.\n\n',
|
||||
'This is perfect for:\n',
|
||||
'- **Event listeners** that need to be re-subscribed when configuration changes\n',
|
||||
'- **Editor decorations** that are replaced when content updates\n',
|
||||
'- **Watchers** that switch targets dynamically\n\n',
|
||||
'```typescript\n',
|
||||
'class MyService extends Disposable {\n',
|
||||
' private readonly _listener = this._register(new MutableDisposable());\n\n',
|
||||
' updateTarget(editor: ICodeEditor): void {\n',
|
||||
' // Old listener is automatically disposed\n',
|
||||
' this._listener.value = editor.onDidChangeModel(() => {\n',
|
||||
' this._handleModelChange();\n',
|
||||
' });\n',
|
||||
' }\n',
|
||||
'}\n',
|
||||
'```\n\n',
|
||||
'The key benefit is that you never forget to dispose the old subscription.\n',
|
||||
], 15)
|
||||
.build(),
|
||||
},
|
||||
// Turn 5: Second user follow-up
|
||||
{
|
||||
kind: 'user',
|
||||
message: 'Can you also show me DisposableMap?',
|
||||
},
|
||||
// Turn 6: Final response
|
||||
{
|
||||
kind: 'content',
|
||||
chunks: new ScenarioBuilder()
|
||||
.wait(20, '## DisposableMap\n\n')
|
||||
.stream([
|
||||
'`DisposableMap<K, V>` extends `Map` with automatic disposal semantics:\n\n',
|
||||
'- When a key is **overwritten**, the old value is disposed\n',
|
||||
'- When a key is **deleted**, the value is disposed\n',
|
||||
'- When the map itself is **disposed**, all values are disposed\n\n',
|
||||
'```typescript\n',
|
||||
'class ToolManager extends Disposable {\n',
|
||||
' private readonly _tools = this._register(new DisposableMap<string, IDisposable>());\n\n',
|
||||
' registerTool(id: string, tool: IDisposable): void {\n',
|
||||
' this._tools.set(id, tool); // auto-disposes previous tool with same id\n',
|
||||
' }\n',
|
||||
'}\n',
|
||||
'```\n\n',
|
||||
'This is commonly used for managing collections of disposable resources keyed by ID.\n',
|
||||
], 15)
|
||||
.build(),
|
||||
},
|
||||
],
|
||||
}),
|
||||
},
|
||||
'long-conversation': {
|
||||
description: '10 user turns, mixed content types',
|
||||
scenario: /** @type {import('./mock-llm-server').MultiTurnScenario} */ ((() => {
|
||||
const topics = [
|
||||
{ question: 'How does the Disposable pattern work?', heading: 'Disposable Pattern', content: 'The `Disposable` base class provides lifecycle management. Subclasses call `this._register()` to track child disposables that are cleaned up automatically when `dispose()` is called.' },
|
||||
{ question: 'What about DisposableStore?', heading: 'DisposableStore', content: '`DisposableStore` aggregates multiple `IDisposable` instances and disposes them all at once. It tracks whether it has already been disposed and throws if you try to add after disposal.' },
|
||||
{ question: 'How does the Event system work?', heading: 'Event System', content: 'The `Emitter<T>` class implements the observer pattern. `Event.once()`, `Event.map()`, `Event.filter()`, and `Event.debounce()` provide functional combinators for composing event streams.' },
|
||||
{ question: 'Explain dependency injection', heading: 'Dependency Injection', content: 'Services are injected through constructor parameters decorated with service identifiers. The `IInstantiationService` resolves dependencies recursively, creating singletons within each scope.' },
|
||||
{ question: 'What is the contribution model?', heading: 'Contribution Model', content: 'Features register functionality through extension points like `Registry.as<IWorkbenchContributionsRegistry>()`. Contributions are instantiated during specific lifecycle phases.' },
|
||||
{ question: 'How does the editor handle text models?', heading: 'Text Models', content: 'The `TextModel` class manages document content with line-based storage. It supports undo/redo stacks, bracket matching, tokenization, and change tracking via edit operations.' },
|
||||
{ question: 'Explain the extension host architecture', heading: 'Extension Host', content: 'Extensions run in a separate process (or worker) called the extension host. Communication happens via an RPC protocol over `IPC`. The main process proxies API calls back to the workbench.' },
|
||||
{ question: 'How does file watching work?', heading: 'File Watching', content: 'The `IFileService` supports correlated and shared file watchers. Correlated watchers are preferred as they track specific resources. The underlying implementation uses `chokidar` or `parcel/watcher`.' },
|
||||
{ question: 'What about the tree widget?', heading: 'Tree Widget', content: 'The `AsyncDataTree` and `ObjectTree` provide virtualized tree rendering. They support filtering, sorting, keyboard navigation, and accessibility. The `ITreeRenderer` interface handles element rendering.' },
|
||||
{ question: 'How does the settings editor work?', heading: 'Settings Editor', content: 'Settings are declared in `package.json` contribution points. The settings editor reads the configuration registry, groups settings by category, and renders appropriate input controls for each type.' },
|
||||
];
|
||||
|
||||
/** @type {import('./mock-llm-server').ScenarioTurn[]} */
|
||||
const turns = [];
|
||||
|
||||
// Turn 1: Initial model response (no user turn needed before the first)
|
||||
const firstTopic = topics[0];
|
||||
turns.push({
|
||||
kind: 'content',
|
||||
chunks: new ScenarioBuilder()
|
||||
.wait(20, `## ${firstTopic.heading}\n\n`)
|
||||
.stream([
|
||||
`${firstTopic.content}\n\n`,
|
||||
'Here is a typical example:\n\n',
|
||||
'```typescript\n',
|
||||
'class MyService extends Disposable {\n',
|
||||
' private readonly _onDidChange = this._register(new Emitter<void>());\n',
|
||||
' readonly onDidChange: Event<void> = this._onDidChange.event;\n\n',
|
||||
' constructor(@IFileService private readonly fileService: IFileService) {\n',
|
||||
' super();\n',
|
||||
' this._register(fileService.onDidFilesChange(e => this._handleChange(e)));\n',
|
||||
' }\n',
|
||||
'}\n',
|
||||
'```\n\n',
|
||||
'Would you like to know more about any specific aspect?\n',
|
||||
], 15)
|
||||
.build(),
|
||||
});
|
||||
|
||||
// Turns 2..N: alternating user follow-up + model response
|
||||
for (let i = 1; i < topics.length; i++) {
|
||||
const topic = topics[i];
|
||||
|
||||
// User follow-up
|
||||
turns.push({ kind: 'user', message: topic.question });
|
||||
|
||||
// Model response — vary content type to stress different renderers
|
||||
const b = new ScenarioBuilder();
|
||||
b.wait(20, `## ${topic.heading}\n\n`);
|
||||
|
||||
// Main explanation
|
||||
const sentences = topic.content.split('. ');
|
||||
b.stream(sentences.map(s => s.endsWith('.') ? s + ' ' : s + '. '), 12);
|
||||
b.emit('\n\n');
|
||||
|
||||
if (i % 3 === 0) {
|
||||
// Every 3rd response: large code block
|
||||
b.emit('```typescript\n');
|
||||
for (let j = 0; j < 8; j++) {
|
||||
b.stream([
|
||||
`export class ${topic.heading.replace(/\s/g, '')}Part${j} extends Disposable {\n`,
|
||||
` private readonly _state = new Map<string, unknown>();\n\n`,
|
||||
` process(input: string): string {\n`,
|
||||
` const cached = this._state.get(input);\n`,
|
||||
` if (cached) { return String(cached); }\n`,
|
||||
` const result = input.split('').reverse().join('');\n`,
|
||||
` this._state.set(input, result);\n`,
|
||||
` return result;\n`,
|
||||
` }\n`,
|
||||
'}\n\n',
|
||||
], 5);
|
||||
}
|
||||
b.emit('```\n\n');
|
||||
} else if (i % 3 === 1) {
|
||||
// Every 3rd+1 response: bullet list with bold + inline code
|
||||
b.emit('Key points to remember:\n\n');
|
||||
for (let j = 0; j < 6; j++) {
|
||||
b.stream([
|
||||
`${j + 1}. **Point ${j + 1}**: The \`${topic.heading.replace(/\s/g, '')}${j}\` `,
|
||||
`component uses the standard pattern with \`_register()\` for lifecycle. `,
|
||||
`It handles edge cases like ${['empty input', 'null references', 'concurrent access', 'circular deps', 'timeout expiry', 'disposal races'][j]}.\n`,
|
||||
], 10);
|
||||
}
|
||||
b.emit('\n');
|
||||
} else {
|
||||
// Every 3rd+2 response: mixed prose + small code snippet
|
||||
b.stream([
|
||||
'This pattern is used extensively throughout the codebase. ',
|
||||
'The key insight is that resources are always tracked from creation, ',
|
||||
'ensuring no leaks even in error paths. ',
|
||||
'The ownership chain is explicit and follows the component hierarchy.\n\n',
|
||||
], 12);
|
||||
b.emit('Quick example:\n\n```typescript\n');
|
||||
b.stream([
|
||||
`const store = new DisposableStore();\n`,
|
||||
`store.add(event.on(() => { /* handler */ }));\n`,
|
||||
`store.add(watcher.watch(uri));\n`,
|
||||
`// Later: store.dispose(); // cleans up everything\n`,
|
||||
], 8);
|
||||
b.emit('```\n\n');
|
||||
}
|
||||
|
||||
b.stream([
|
||||
`That covers the essentials of **${topic.heading}**. `,
|
||||
'Let me know if you want to dive deeper into any of these concepts.\n',
|
||||
], 15);
|
||||
|
||||
turns.push({
|
||||
kind: 'content',
|
||||
chunks: b.build(),
|
||||
});
|
||||
}
|
||||
|
||||
return { type: 'multi-turn', turns };
|
||||
})()),
|
||||
},
|
||||
};
|
||||
|
||||
// -- Registration helper ------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Get a brief description of a scenario by ID.
|
||||
* @param {string} id
|
||||
* @returns {string}
|
||||
*/
|
||||
function getScenarioDescription(id) {
|
||||
const content = CONTENT_SCENARIOS[id];
|
||||
if (content) { return content.description; }
|
||||
const tool = TOOL_CALL_SCENARIOS[id];
|
||||
if (tool) { return tool.description; }
|
||||
const multi = MULTI_TURN_SCENARIOS[id];
|
||||
if (multi) { return multi.description; }
|
||||
return '';
|
||||
}
|
||||
|
||||
/**
|
||||
* Register all built-in perf scenarios into the mock LLM server.
|
||||
* Call this from your test file before starting the server.
|
||||
*/
|
||||
function registerPerfScenarios() {
|
||||
for (const [id, def] of Object.entries(CONTENT_SCENARIOS)) {
|
||||
registerScenario(id, def.chunks);
|
||||
}
|
||||
for (const [id, def] of Object.entries(TOOL_CALL_SCENARIOS)) {
|
||||
registerScenario(id, def.scenario);
|
||||
}
|
||||
for (const [id, def] of Object.entries(MULTI_TURN_SCENARIOS)) {
|
||||
registerScenario(id, def.scenario);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { registerPerfScenarios, getScenarioDescription, CONTENT_SCENARIOS, TOOL_CALL_SCENARIOS, MULTI_TURN_SCENARIOS };
|
||||
@@ -0,0 +1,835 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
// @ts-check
|
||||
|
||||
/**
|
||||
* Shared utilities for chat performance benchmarks and leak checks.
|
||||
*
|
||||
* Platform: macOS and Linux only. Windows is not supported — several
|
||||
* utilities (`sqlite3`, `sleep`, `pkill`) are Unix-specific.
|
||||
* CI runs on ubuntu-latest.
|
||||
*/
|
||||
|
||||
const path = require('path');
|
||||
const fs = require('fs');
|
||||
const os = require('os');
|
||||
const http = require('http');
|
||||
const { execSync, execFileSync, spawn } = require('child_process');
|
||||
|
||||
const ROOT = path.join(__dirname, '..', '..', '..');
|
||||
const DATA_DIR = path.join(ROOT, '.chat-simulation-data');
|
||||
|
||||
// -- Config loading ----------------------------------------------------------
|
||||
|
||||
/** @param {string} text */
|
||||
function stripJsoncComments(text) { return text.replace(/\/\/.*/g, '').replace(/\/\*[\s\S]*?\*\//g, ''); }
|
||||
|
||||
/**
|
||||
* Load a namespaced section from config.jsonc.
|
||||
* @param {string} section - Top-level key (e.g. 'perfRegression', 'memLeaks')
|
||||
* @returns {Record<string, any>}
|
||||
*/
|
||||
function loadConfig(section) {
|
||||
const raw = fs.readFileSync(path.join(__dirname, '..', 'config.jsonc'), 'utf-8');
|
||||
const config = JSON.parse(stripJsoncComments(raw));
|
||||
return config[section] ?? {};
|
||||
}
|
||||
|
||||
// -- Electron path resolution ------------------------------------------------
|
||||
|
||||
/**
|
||||
* Derive the VS Code repo root from an Electron executable path.
|
||||
* Dev builds live at `<repo>/.build/electron/<app>/`, so we walk up
|
||||
* from the path to find the directory containing `.build`.
|
||||
* Returns `undefined` if the path doesn't look like a dev build.
|
||||
* @param {string} electronPath
|
||||
* @returns {string | undefined}
|
||||
*/
|
||||
function getRepoRoot(electronPath) {
|
||||
const buildIdx = electronPath.indexOf(`${path.sep}.build${path.sep}`);
|
||||
if (buildIdx === -1) {
|
||||
// Also check for posix separators (path may be user-supplied)
|
||||
const posixIdx = electronPath.indexOf('/.build/');
|
||||
if (posixIdx === -1) { return undefined; }
|
||||
return electronPath.slice(0, posixIdx);
|
||||
}
|
||||
return electronPath.slice(0, buildIdx);
|
||||
}
|
||||
|
||||
function getElectronPath() {
|
||||
const product = require(path.join(ROOT, 'product.json'));
|
||||
if (process.platform === 'darwin') {
|
||||
return path.join(ROOT, '.build', 'electron', `${product.nameLong}.app`, 'Contents', 'MacOS', product.nameShort);
|
||||
} else if (process.platform === 'linux') {
|
||||
return path.join(ROOT, '.build', 'electron', product.applicationName);
|
||||
} else {
|
||||
return path.join(ROOT, '.build', 'electron', `${product.nameShort}.exe`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns true if the string looks like a VS Code version or commit hash
|
||||
* rather than a file path.
|
||||
* @param {string} value
|
||||
*/
|
||||
function isVersionString(value) {
|
||||
if (value === 'insiders' || value === 'stable') { return true; }
|
||||
if (/^\d+\.\d+\.\d+/.test(value)) { return true; }
|
||||
if (/^[0-9a-f]{7,40}$/.test(value)) { return true; }
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the built-in extensions directory for a VS Code executable.
|
||||
* @param {string} exePath
|
||||
* @returns {string | undefined}
|
||||
*/
|
||||
function getBuiltinExtensionsDir(exePath) {
|
||||
if (process.platform === 'darwin') {
|
||||
const appDir = exePath.split('/Contents/')[0];
|
||||
return path.join(appDir, 'Contents', 'Resources', 'app', 'extensions');
|
||||
} else if (process.platform === 'linux') {
|
||||
return path.join(path.dirname(exePath), 'resources', 'app', 'extensions');
|
||||
} else {
|
||||
return path.join(path.dirname(exePath), 'resources', 'app', 'extensions');
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve a build arg to an executable path.
|
||||
* Version strings are downloaded via @vscode/test-electron.
|
||||
* @param {string | undefined} buildArg
|
||||
* @returns {Promise<string>}
|
||||
*/
|
||||
async function resolveBuild(buildArg) {
|
||||
if (!buildArg) {
|
||||
return getElectronPath();
|
||||
}
|
||||
if (isVersionString(buildArg)) {
|
||||
console.log(`[chat-simulation] Downloading VS Code ${buildArg}...`);
|
||||
const { downloadAndUnzipVSCode, resolveCliArgsFromVSCodeExecutablePath } = require('@vscode/test-electron');
|
||||
const exePath = await downloadAndUnzipVSCode(buildArg);
|
||||
console.log(`[chat-simulation] Downloaded: ${exePath}`);
|
||||
|
||||
// Check if copilot is already bundled as a built-in extension
|
||||
// (recent Insiders/Stable builds ship it in the app's extensions/ dir).
|
||||
const builtinExtDir = getBuiltinExtensionsDir(exePath);
|
||||
const hasCopilotBuiltin = builtinExtDir && fs.existsSync(builtinExtDir)
|
||||
&& fs.readdirSync(builtinExtDir).some(e => e === 'copilot');
|
||||
|
||||
if (hasCopilotBuiltin) {
|
||||
console.log(`[chat-simulation] Copilot is bundled as a built-in extension`);
|
||||
} else {
|
||||
// Install copilot-chat from the marketplace into our shared
|
||||
// extensions dir so it's available when we launch with
|
||||
// --extensions-dir=DATA_DIR/extensions.
|
||||
const extDir = path.join(DATA_DIR, 'extensions');
|
||||
fs.mkdirSync(extDir, { recursive: true });
|
||||
const [cli, ...cliArgs] = resolveCliArgsFromVSCodeExecutablePath(exePath);
|
||||
const extId = 'GitHub.copilot-chat';
|
||||
console.log(`[chat-simulation] Installing ${extId} into ${extDir}...`);
|
||||
const { spawnSync } = require('child_process');
|
||||
const result = spawnSync(cli, [...cliArgs, '--extensions-dir', extDir, '--install-extension', extId], {
|
||||
encoding: 'utf-8',
|
||||
stdio: 'pipe',
|
||||
shell: process.platform === 'win32',
|
||||
timeout: 120_000,
|
||||
});
|
||||
if (result.status !== 0) {
|
||||
console.warn(`[chat-simulation] Extension install exited with ${result.status}: ${(result.stderr || '').substring(0, 500)}`);
|
||||
} else {
|
||||
console.log(`[chat-simulation] ${extId} installed`);
|
||||
}
|
||||
}
|
||||
|
||||
return exePath;
|
||||
}
|
||||
return path.resolve(buildArg);
|
||||
}
|
||||
|
||||
// -- Storage pre-seeding -----------------------------------------------------
|
||||
|
||||
/**
|
||||
* Pre-seed the VS Code storage database to prevent the
|
||||
* BuiltinChatExtensionEnablementMigration from disabling the copilot
|
||||
* extension on fresh user data directories.
|
||||
*
|
||||
* Requires `sqlite3` on PATH (pre-installed on macOS and Ubuntu).
|
||||
* @param {string} userDataDir
|
||||
*/
|
||||
function preseedStorage(userDataDir) {
|
||||
const globalStorageDir = path.join(userDataDir, 'User', 'globalStorage');
|
||||
fs.mkdirSync(globalStorageDir, { recursive: true });
|
||||
const dbPath = path.join(globalStorageDir, 'state.vscdb');
|
||||
const sql = [
|
||||
'CREATE TABLE IF NOT EXISTS ItemTable (key TEXT UNIQUE ON CONFLICT REPLACE, value BLOB);',
|
||||
'INSERT INTO ItemTable (key, value) VALUES (\'builtinChatExtensionEnablementMigration\', \'true\');',
|
||||
'INSERT INTO ItemTable (key, value) VALUES (\'chat.tools.global.autoApprove.optIn\', \'true\');',
|
||||
].join(' ');
|
||||
execFileSync('sqlite3', [dbPath, sql]);
|
||||
}
|
||||
|
||||
// -- Launch helpers ----------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Build the environment variables for launching VS Code with the mock server.
|
||||
* @param {{ url: string }} mockServer
|
||||
* @param {{ isDevBuild?: boolean }} [opts]
|
||||
* @returns {Record<string, string>}
|
||||
*/
|
||||
function buildEnv(mockServer, { isDevBuild = true } = {}) {
|
||||
/** @type {Record<string, string>} */
|
||||
const env = {
|
||||
...process.env,
|
||||
ELECTRON_ENABLE_LOGGING: '1',
|
||||
IS_SCENARIO_AUTOMATION: '1',
|
||||
GITHUB_PAT: 'perf-benchmark-fake-pat',
|
||||
VSCODE_COPILOT_CHAT_TOKEN: Buffer.from(JSON.stringify({
|
||||
token: 'perf-benchmark-fake-token',
|
||||
expires_at: Math.floor(Date.now() / 1000) + 3600,
|
||||
refresh_in: 1800,
|
||||
sku: 'free_limited_copilot',
|
||||
individual: true,
|
||||
isNoAuthUser: true,
|
||||
copilot_plan: 'free',
|
||||
organization_login_list: [],
|
||||
endpoints: { api: mockServer.url, proxy: mockServer.url },
|
||||
})).toString('base64'),
|
||||
};
|
||||
// Dev-only flags — these tell Electron to load the app from source (out/)
|
||||
// instead of the packaged app. Setting them on a stable build causes it
|
||||
// to fail to show a window.
|
||||
if (isDevBuild) {
|
||||
env.NODE_ENV = 'development';
|
||||
env.VSCODE_DEV = '1';
|
||||
env.VSCODE_CLI = '1';
|
||||
}
|
||||
return env;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the default VS Code launch args.
|
||||
* @param {string} userDataDir
|
||||
* @param {string} extDir
|
||||
* @param {string} logsDir
|
||||
* @returns {string[]}
|
||||
*/
|
||||
function buildArgs(userDataDir, extDir, logsDir, { isDevBuild = true, extHostInspectPort = 0, traceFile = '', appRoot = ROOT } = {}) {
|
||||
// Chromium switches must come BEFORE the app path (ROOT) — Chromium
|
||||
// only processes switches that precede the first non-switch argument.
|
||||
const chromiumFlags = [];
|
||||
if (traceFile) {
|
||||
chromiumFlags.push(`--enable-tracing=v8.gc,devtools.timeline,blink.user_timing`);
|
||||
chromiumFlags.push(`--trace-startup-file=${traceFile}`);
|
||||
chromiumFlags.push(`--enable-tracing-format=json`);
|
||||
}
|
||||
const args = [
|
||||
...chromiumFlags,
|
||||
appRoot,
|
||||
'--skip-release-notes',
|
||||
'--skip-welcome',
|
||||
'--disable-telemetry',
|
||||
'--disable-updates',
|
||||
'--disable-workspace-trust',
|
||||
`--user-data-dir=${userDataDir}`,
|
||||
`--extensions-dir=${extDir}`,
|
||||
`--logsPath=${logsDir}`,
|
||||
'--enable-smoke-test-driver',
|
||||
'--disable-extensions',
|
||||
];
|
||||
// vscode-api-tests only exists in the dev build
|
||||
if (isDevBuild) {
|
||||
args.push('--disable-extension=vscode.vscode-api-tests');
|
||||
}
|
||||
if (process.platform !== 'darwin') {
|
||||
args.push('--disable-gpu');
|
||||
}
|
||||
if (process.env.CI && process.platform === 'linux') {
|
||||
args.push('--no-sandbox');
|
||||
}
|
||||
// Enable extension host inspector for profiling/heap snapshots
|
||||
if (extHostInspectPort > 0) {
|
||||
args.push(`--inspect-extensions=${extHostInspectPort}`);
|
||||
}
|
||||
return args;
|
||||
}
|
||||
|
||||
/**
|
||||
* Write VS Code settings that point the copilot extension at the mock server.
|
||||
* @param {string} userDataDir
|
||||
* @param {{ url: string }} mockServer
|
||||
* @param {Record<string, any>} [overrides]
|
||||
*/
|
||||
function writeSettings(userDataDir, mockServer, overrides) {
|
||||
const settingsDir = path.join(userDataDir, 'User');
|
||||
fs.mkdirSync(settingsDir, { recursive: true });
|
||||
fs.writeFileSync(path.join(settingsDir, 'settings.json'), JSON.stringify({
|
||||
'github.copilot.advanced.debug.overrideProxyUrl': mockServer.url,
|
||||
'github.copilot.advanced.debug.overrideCapiUrl': mockServer.url,
|
||||
'chat.allowAnonymousAccess': true,
|
||||
// Disable MCP servers — they start async and add unpredictable
|
||||
// delay that pollutes perf measurements.
|
||||
'chat.mcp.discovery.enabled': false,
|
||||
'chat.mcp.enabled': false,
|
||||
'github.copilot.chat.githubMcpServer.enabled': false,
|
||||
'github.copilot.chat.cli.mcp.enabled': false,
|
||||
// Auto-approve all tool invocations (YOLO mode) so tool call
|
||||
// scenarios don't block on confirmation dialogs.
|
||||
'chat.tools.global.autoApprove': true,
|
||||
...overrides,
|
||||
}, null, '\t'));
|
||||
}
|
||||
|
||||
/**
|
||||
* Prepare a fresh run directory (clean, create, preseed, write settings).
|
||||
* @param {string} runId
|
||||
* @param {{ url: string }} mockServer
|
||||
* @param {Record<string, any>} [settingsOverrides]
|
||||
* @returns {{ userDataDir: string, extDir: string, logsDir: string }}
|
||||
*/
|
||||
function prepareRunDir(runId, mockServer, settingsOverrides) {
|
||||
const tmpBase = path.join(os.tmpdir(), 'vscode-chat-simulation');
|
||||
const userDataDir = path.join(tmpBase, `run-${runId}`);
|
||||
const extDir = path.join(DATA_DIR, 'extensions');
|
||||
const logsDir = path.join(tmpBase, 'logs', `run-${runId}`);
|
||||
// Retry rmSync to handle ENOTEMPTY race conditions from Electron cache locks
|
||||
for (let attempt = 0; attempt < 3; attempt++) {
|
||||
try {
|
||||
fs.rmSync(userDataDir, { recursive: true, force: true });
|
||||
break;
|
||||
} catch (err) {
|
||||
const error = /** @type {NodeJS.ErrnoException} */ (err);
|
||||
if (attempt < 2 && error.code === 'ENOTEMPTY') {
|
||||
require('child_process').execSync(`sleep 0.5`);
|
||||
} else {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
fs.mkdirSync(userDataDir, { recursive: true });
|
||||
fs.mkdirSync(extDir, { recursive: true });
|
||||
fs.mkdirSync(logsDir, { recursive: true });
|
||||
preseedStorage(userDataDir);
|
||||
writeSettings(userDataDir, mockServer, settingsOverrides);
|
||||
return { userDataDir, extDir, logsDir };
|
||||
}
|
||||
|
||||
// -- VS Code launch via CDP --------------------------------------------------
|
||||
|
||||
// -- Extension host inspector ------------------------------------------------
|
||||
|
||||
/** @type {number} */
|
||||
let nextExtHostPort = 29222;
|
||||
|
||||
/** @returns {number} */
|
||||
function getNextExtHostInspectPort() {
|
||||
return nextExtHostPort++;
|
||||
}
|
||||
|
||||
/**
|
||||
* Connect to the extension host's Node inspector via WebSocket.
|
||||
* The extension host must be started with `--inspect-extensions=<port>`.
|
||||
*
|
||||
* @param {number} port
|
||||
* @param {{ verbose?: boolean, timeoutMs?: number }} [opts]
|
||||
* @returns {Promise<{ send: (method: string, params?: any) => Promise<any>, on: (event: string, listener: (params: any) => void) => void, close: () => void, port: number }>}
|
||||
*/
|
||||
async function connectToExtHostInspector(port, opts = {}) {
|
||||
const { verbose = false, timeoutMs = 30_000 } = opts;
|
||||
|
||||
// Wait for the inspector endpoint to be available
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
/** @type {any} */
|
||||
let wsUrl;
|
||||
while (Date.now() < deadline) {
|
||||
try {
|
||||
const targets = await getJson(`http://127.0.0.1:${port}/json`);
|
||||
if (targets.length > 0 && targets[0].webSocketDebuggerUrl) {
|
||||
wsUrl = targets[0].webSocketDebuggerUrl;
|
||||
break;
|
||||
}
|
||||
} catch { }
|
||||
await new Promise(r => setTimeout(r, 500));
|
||||
}
|
||||
if (!wsUrl) {
|
||||
throw new Error(`Timed out waiting for extension host inspector on port ${port}`);
|
||||
}
|
||||
|
||||
if (verbose) {
|
||||
console.log(` [ext-host] Connected to inspector: ${wsUrl}`);
|
||||
}
|
||||
|
||||
const WebSocket = require('ws');
|
||||
const ws = new WebSocket(wsUrl);
|
||||
await new Promise((resolve, reject) => {
|
||||
ws.once('open', resolve);
|
||||
ws.once('error', reject);
|
||||
});
|
||||
|
||||
let msgId = 1;
|
||||
/** @type {Map<number, { resolve: (v: any) => void, reject: (e: Error) => void }>} */
|
||||
const pending = new Map();
|
||||
/** @type {Map<string, ((params: any) => void)[]>} */
|
||||
const eventListeners = new Map();
|
||||
|
||||
ws.on('message', (/** @type {Buffer} */ data) => {
|
||||
const msg = JSON.parse(data.toString());
|
||||
if (msg.id !== undefined) {
|
||||
const p = pending.get(msg.id);
|
||||
if (p) {
|
||||
pending.delete(msg.id);
|
||||
if (msg.error) { p.reject(new Error(msg.error.message)); }
|
||||
else { p.resolve(msg.result); }
|
||||
}
|
||||
} else if (msg.method) {
|
||||
const listeners = eventListeners.get(msg.method) || [];
|
||||
for (const listener of listeners) { listener(msg.params); }
|
||||
}
|
||||
});
|
||||
|
||||
return {
|
||||
port,
|
||||
/**
|
||||
* @param {string} method
|
||||
* @param {any} [params]
|
||||
* @returns {Promise<any>}
|
||||
*/
|
||||
send(method, params) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const id = msgId++;
|
||||
pending.set(id, { resolve, reject });
|
||||
ws.send(JSON.stringify({ id, method, params }));
|
||||
setTimeout(() => {
|
||||
if (pending.has(id)) {
|
||||
pending.delete(id);
|
||||
reject(new Error(`Inspector call timed out: ${method}`));
|
||||
}
|
||||
}, 30_000);
|
||||
});
|
||||
},
|
||||
/**
|
||||
* @param {string} event
|
||||
* @param {(params: any) => void} listener
|
||||
*/
|
||||
on(event, listener) {
|
||||
const list = eventListeners.get(event) || [];
|
||||
list.push(listener);
|
||||
eventListeners.set(event, list);
|
||||
},
|
||||
close() {
|
||||
ws.close();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch JSON from a URL. Used to probe the CDP endpoint.
|
||||
* @param {string} url
|
||||
* @returns {Promise<any>}
|
||||
*/
|
||||
function getJson(url) {
|
||||
return new Promise((resolve, reject) => {
|
||||
http.get(url, res => {
|
||||
let data = '';
|
||||
res.on('data', chunk => { data += chunk; });
|
||||
res.on('end', () => {
|
||||
try { resolve(JSON.parse(data)); }
|
||||
catch { reject(new Error(`Invalid JSON from ${url}`)); }
|
||||
});
|
||||
}).on('error', reject);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait until VS Code exposes its CDP endpoint.
|
||||
* @param {number} port
|
||||
* @param {number} timeoutMs
|
||||
* @returns {Promise<void>}
|
||||
*/
|
||||
async function waitForCDP(port, timeoutMs = 60_000) {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
try {
|
||||
await getJson(`http://127.0.0.1:${port}/json/version`);
|
||||
return;
|
||||
} catch {
|
||||
await new Promise(r => setTimeout(r, 500));
|
||||
}
|
||||
}
|
||||
throw new Error(`Timed out waiting for CDP on port ${port}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the workbench page among all CDP pages.
|
||||
* For dev builds this checks for `globalThis.driver` (smoke-test driver).
|
||||
* For stable builds it checks for `.monaco-workbench` in the DOM.
|
||||
* @param {import('playwright').Browser} browser
|
||||
* @param {number} timeoutMs
|
||||
* @returns {Promise<import('playwright').Page>}
|
||||
*/
|
||||
async function findWorkbenchPage(browser, timeoutMs = 60_000) {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
const pages = browser.contexts().flatMap(ctx => ctx.pages());
|
||||
for (const page of pages) {
|
||||
const hasWorkbench = await page.evaluate(() =>
|
||||
// @ts-ignore
|
||||
!!globalThis.driver?.whenWorkbenchRestored || !!document.querySelector('.monaco-workbench')
|
||||
).catch(() => false);
|
||||
if (hasWorkbench) {
|
||||
return page;
|
||||
}
|
||||
}
|
||||
await new Promise(r => setTimeout(r, 500));
|
||||
}
|
||||
throw new Error('Timed out waiting for the workbench page');
|
||||
}
|
||||
|
||||
/** @type {number} */
|
||||
let nextPort = 19222;
|
||||
|
||||
/**
|
||||
* Launch VS Code via child_process and connect via CDP.
|
||||
* Works with dev builds, insiders, and stable releases.
|
||||
*
|
||||
* @param {string} executable - Path to the VS Code executable (Electron binary or CLI)
|
||||
* @param {string[]} launchArgs - Arguments to pass to the executable
|
||||
* @param {Record<string, string>} env - Environment variables
|
||||
* @param {{ verbose?: boolean }} [opts]
|
||||
* @returns {Promise<{ page: import('playwright').Page, browser: import('playwright').Browser, close: () => Promise<void> }>}
|
||||
*/
|
||||
async function launchVSCode(executable, launchArgs, env, opts = {}) {
|
||||
const { chromium } = require('playwright');
|
||||
const port = nextPort++;
|
||||
|
||||
const args = [`--remote-debugging-port=${port}`, ...launchArgs];
|
||||
const isShell = process.platform === 'win32';
|
||||
|
||||
if (opts.verbose) {
|
||||
console.log(` [launch] ${executable} ${args.slice(0, 3).join(' ')} ... (port ${port})`);
|
||||
}
|
||||
|
||||
const child = spawn(executable, args, {
|
||||
cwd: ROOT,
|
||||
env,
|
||||
shell: isShell,
|
||||
stdio: opts.verbose ? 'inherit' : ['ignore', 'ignore', 'ignore'],
|
||||
});
|
||||
|
||||
// Track early exit
|
||||
let exitError = /** @type {Error | null} */ (null);
|
||||
child.once('exit', (code, signal) => {
|
||||
if (!exitError) {
|
||||
exitError = new Error(`VS Code exited before CDP connected (code=${code} signal=${signal})`);
|
||||
}
|
||||
});
|
||||
|
||||
// Wait for CDP
|
||||
try {
|
||||
await waitForCDP(port);
|
||||
} catch (e) {
|
||||
if (exitError) { throw exitError; }
|
||||
throw e;
|
||||
}
|
||||
|
||||
const browser = await chromium.connectOverCDP(`http://127.0.0.1:${port}`);
|
||||
const page = await findWorkbenchPage(browser);
|
||||
|
||||
return {
|
||||
page,
|
||||
browser,
|
||||
close: async () => {
|
||||
// Trigger app.quit() so Chromium flushes trace buffers and
|
||||
// writes --trace-startup-file. Using Cmd+Q / Alt+F4 triggers
|
||||
// the full Electron quit lifecycle including trace flush.
|
||||
// window.close() only closes the BrowserWindow without
|
||||
// triggering app-level quit.
|
||||
try {
|
||||
const quitKey = process.platform === 'darwin' ? 'Meta+KeyQ' : 'Alt+F4';
|
||||
await page.keyboard.press(quitKey);
|
||||
} catch {
|
||||
// Page may already be closed
|
||||
}
|
||||
const pid = child.pid;
|
||||
// Wait for graceful exit (up to 30s for trace flush)
|
||||
await new Promise(resolve => {
|
||||
const timer = setTimeout(() => {
|
||||
if (pid) {
|
||||
try { execSync(`pkill -9 -P ${pid}`, { stdio: 'ignore' }); }
|
||||
catch { }
|
||||
}
|
||||
child.kill('SIGKILL');
|
||||
resolve(undefined);
|
||||
}, 30_000);
|
||||
child.once('exit', () => { clearTimeout(timer); resolve(undefined); });
|
||||
});
|
||||
// Disconnect CDP after the process has exited
|
||||
await browser.close().catch(() => { });
|
||||
// Kill crashpad handler — it self-daemonizes and outlives the
|
||||
// parent. Wait briefly for it to detach, then kill by pattern.
|
||||
await new Promise(r => setTimeout(r, 500));
|
||||
try { execSync('pkill -9 -f crashpad_handler.*vscode-chat-simulation', { stdio: 'ignore' }); }
|
||||
catch { }
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// -- Statistics --------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* @param {number[]} values
|
||||
*/
|
||||
function median(values) {
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
const mid = Math.floor(sorted.length / 2);
|
||||
return sorted.length % 2 !== 0 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2;
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove outliers using IQR method.
|
||||
* @param {number[]} values
|
||||
* @returns {number[]}
|
||||
*/
|
||||
function removeOutliers(values) {
|
||||
if (values.length < 4) { return values; }
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
const q1 = sorted[Math.floor(sorted.length * 0.25)];
|
||||
const q3 = sorted[Math.floor(sorted.length * 0.75)];
|
||||
const iqr = q3 - q1;
|
||||
const lo = q1 - 1.5 * iqr;
|
||||
const hi = q3 + 1.5 * iqr;
|
||||
return sorted.filter(v => v >= lo && v <= hi);
|
||||
}
|
||||
|
||||
/**
|
||||
* Regularized incomplete beta function I_x(a, b) via continued fraction.
|
||||
* Used for computing t-distribution CDF / p-values.
|
||||
* @param {number} x
|
||||
* @param {number} a
|
||||
* @param {number} b
|
||||
* @returns {number}
|
||||
*/
|
||||
function betaIncomplete(x, a, b) {
|
||||
if (x <= 0) { return 0; }
|
||||
if (x >= 1) { return 1; }
|
||||
// Use symmetry relation when x > (a+1)/(a+b+2) for better convergence
|
||||
if (x > (a + 1) / (a + b + 2)) {
|
||||
return 1 - betaIncomplete(1 - x, b, a);
|
||||
}
|
||||
// Log-beta via Stirling: lnBeta(a,b) = lnGamma(a)+lnGamma(b)-lnGamma(a+b)
|
||||
const lnBeta = lnGamma(a) + lnGamma(b) - lnGamma(a + b);
|
||||
const front = Math.exp(Math.log(x) * a + Math.log(1 - x) * b - lnBeta) / a;
|
||||
// Lentz's continued fraction
|
||||
const maxIter = 200;
|
||||
const eps = 1e-14;
|
||||
let c = 1, d = 1 - (a + b) * x / (a + 1);
|
||||
if (Math.abs(d) < eps) { d = eps; }
|
||||
d = 1 / d;
|
||||
let result = d;
|
||||
for (let m = 1; m <= maxIter; m++) {
|
||||
// Even step
|
||||
let num = m * (b - m) * x / ((a + 2 * m - 1) * (a + 2 * m));
|
||||
d = 1 + num * d; if (Math.abs(d) < eps) { d = eps; } d = 1 / d;
|
||||
c = 1 + num / c; if (Math.abs(c) < eps) { c = eps; }
|
||||
result *= d * c;
|
||||
// Odd step
|
||||
num = -(a + m) * (a + b + m) * x / ((a + 2 * m) * (a + 2 * m + 1));
|
||||
d = 1 + num * d; if (Math.abs(d) < eps) { d = eps; } d = 1 / d;
|
||||
c = 1 + num / c; if (Math.abs(c) < eps) { c = eps; }
|
||||
const delta = d * c;
|
||||
result *= delta;
|
||||
if (Math.abs(delta - 1) < eps) { break; }
|
||||
}
|
||||
return front * result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Log-gamma via Lanczos approximation.
|
||||
* @param {number} z
|
||||
* @returns {number}
|
||||
*/
|
||||
function lnGamma(z) {
|
||||
const g = 7;
|
||||
const coef = [0.99999999999980993, 676.5203681218851, -1259.1392167224028,
|
||||
771.32342877765313, -176.61502916214059, 12.507343278686905,
|
||||
-0.13857109526572012, 9.9843695780195716e-6, 1.5056327351493116e-7];
|
||||
if (z < 0.5) {
|
||||
return Math.log(Math.PI / Math.sin(Math.PI * z)) - lnGamma(1 - z);
|
||||
}
|
||||
z -= 1;
|
||||
let x = coef[0];
|
||||
for (let i = 1; i < g + 2; i++) { x += coef[i] / (z + i); }
|
||||
const t = z + g + 0.5;
|
||||
return 0.5 * Math.log(2 * Math.PI) + (z + 0.5) * Math.log(t) - t + Math.log(x);
|
||||
}
|
||||
|
||||
/**
|
||||
* Two-tailed p-value from t-distribution.
|
||||
* @param {number} t - t-statistic
|
||||
* @param {number} df - degrees of freedom
|
||||
* @returns {number}
|
||||
*/
|
||||
function tDistPValue(t, df) {
|
||||
const x = df / (df + t * t);
|
||||
return betaIncomplete(x, df / 2, 0.5);
|
||||
}
|
||||
|
||||
/**
|
||||
* Welch's t-test for two independent samples (unequal variance).
|
||||
* @param {number[]} a - Sample 1 (e.g., baseline values)
|
||||
* @param {number[]} b - Sample 2 (e.g., current values)
|
||||
* @returns {{ t: number, df: number, pValue: number, significant: boolean, confidence: string } | null}
|
||||
*/
|
||||
function welchTTest(a, b) {
|
||||
if (a.length < 2 || b.length < 2) { return null; }
|
||||
const meanA = a.reduce((s, v) => s + v, 0) / a.length;
|
||||
const meanB = b.reduce((s, v) => s + v, 0) / b.length;
|
||||
const varA = a.reduce((s, v) => s + (v - meanA) ** 2, 0) / (a.length - 1);
|
||||
const varB = b.reduce((s, v) => s + (v - meanB) ** 2, 0) / (b.length - 1);
|
||||
const seA = varA / a.length;
|
||||
const seB = varB / b.length;
|
||||
const seDiff = Math.sqrt(seA + seB);
|
||||
if (seDiff === 0) { return null; }
|
||||
const t = (meanB - meanA) / seDiff;
|
||||
// Welch-Satterthwaite degrees of freedom
|
||||
const df = (seA + seB) ** 2 / ((seA ** 2) / (a.length - 1) + (seB ** 2) / (b.length - 1));
|
||||
const pValue = tDistPValue(t, df);
|
||||
const significant = pValue < 0.05;
|
||||
let confidence;
|
||||
if (pValue < 0.01) { confidence = 'high'; }
|
||||
else if (pValue < 0.05) { confidence = 'medium'; }
|
||||
else if (pValue < 0.1) { confidence = 'low'; }
|
||||
else { confidence = 'none'; }
|
||||
return { t: Math.round(t * 100) / 100, df: Math.round(df * 10) / 10, pValue: Math.round(pValue * 1000) / 1000, significant, confidence };
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute robust stats for a metric array.
|
||||
* @param {number[]} raw
|
||||
*/
|
||||
function robustStats(raw) {
|
||||
const valid = raw.filter(v => v >= 0);
|
||||
if (valid.length === 0) { return null; }
|
||||
const cleaned = removeOutliers(valid);
|
||||
if (cleaned.length === 0) { return null; }
|
||||
const sorted = [...cleaned].sort((a, b) => a - b);
|
||||
const med = median(sorted);
|
||||
const p95 = sorted[Math.min(Math.floor(sorted.length * 0.95), sorted.length - 1)];
|
||||
const mean = sorted.reduce((a, b) => a + b, 0) / sorted.length;
|
||||
const variance = sorted.reduce((a, b) => a + (b - mean) ** 2, 0) / sorted.length;
|
||||
const stddev = Math.sqrt(variance);
|
||||
const cv = mean > 0 ? stddev / mean : 0;
|
||||
return {
|
||||
median: Math.round(med * 100) / 100,
|
||||
p95: Math.round(p95 * 100) / 100,
|
||||
min: sorted[0],
|
||||
max: sorted[sorted.length - 1],
|
||||
mean: Math.round(mean * 100) / 100,
|
||||
stddev: Math.round(stddev * 100) / 100,
|
||||
cv: Math.round(cv * 1000) / 1000,
|
||||
n: sorted.length,
|
||||
nOutliers: valid.length - cleaned.length,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Simple linear regression slope (y per unit x).
|
||||
* @param {number[]} values
|
||||
*/
|
||||
function linearRegressionSlope(values) {
|
||||
const n = values.length;
|
||||
if (n < 2) { return 0; }
|
||||
let sumX = 0, sumY = 0, sumXY = 0, sumX2 = 0;
|
||||
for (let i = 0; i < n; i++) {
|
||||
sumX += i;
|
||||
sumY += values[i];
|
||||
sumXY += i * values[i];
|
||||
sumX2 += i * i;
|
||||
}
|
||||
return (n * sumXY - sumX * sumY) / (n * sumX2 - sumX * sumX);
|
||||
}
|
||||
|
||||
/**
|
||||
* Format a single metric line for console output.
|
||||
* @param {number[]} values
|
||||
* @param {string} label
|
||||
* @param {string} unit
|
||||
*/
|
||||
function summarize(values, label, unit) {
|
||||
const s = robustStats(values);
|
||||
if (!s) { return ` ${label}: (no data)`; }
|
||||
const cv = s.cv > 0.15 ? ` cv=${(s.cv * 100).toFixed(0)}%⚠` : ` cv=${(s.cv * 100).toFixed(0)}%`;
|
||||
const outliers = s.nOutliers > 0 ? ` (${s.nOutliers} outlier${s.nOutliers > 1 ? 's' : ''} removed)` : '';
|
||||
return ` ${label}: median=${s.median}${unit}, p95=${s.p95}${unit},${cv}${outliers} [n=${s.n}]`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute duration between two chat perf marks.
|
||||
* @param {Array<{name: string, startTime: number}>} marks
|
||||
* @param {string} from
|
||||
* @param {string} to
|
||||
*/
|
||||
function markDuration(marks, from, to) {
|
||||
const fromMark = marks.find(m => m.name.endsWith('/' + from));
|
||||
const toMark = marks.find(m => m.name.endsWith('/' + to));
|
||||
if (fromMark && toMark) {
|
||||
return toMark.startTime - fromMark.startTime;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
/** @type {Array<[string, string, string]>} */
|
||||
const METRIC_DEFS = [
|
||||
['timeToFirstToken', 'timing', 'ms'],
|
||||
['timeToComplete', 'timing', 'ms'],
|
||||
['timeToRenderComplete', 'timing', 'ms'],
|
||||
['timeToUIUpdated', 'timing', 'ms'],
|
||||
['instructionCollectionTime', 'timing', 'ms'],
|
||||
['agentInvokeTime', 'timing', 'ms'],
|
||||
['heapDelta', 'memory', 'MB'],
|
||||
['heapDeltaPostGC', 'memory', 'MB'],
|
||||
['gcDurationMs', 'memory', 'ms'],
|
||||
['layoutCount', 'rendering', ''],
|
||||
['layoutDurationMs', 'rendering', 'ms'],
|
||||
['recalcStyleCount', 'rendering', ''],
|
||||
['forcedReflowCount', 'rendering', ''],
|
||||
['longTaskCount', 'rendering', ''],
|
||||
['longAnimationFrameCount', 'rendering', ''],
|
||||
['longAnimationFrameTotalMs', 'rendering', 'ms'],
|
||||
['frameCount', 'rendering', ''],
|
||||
['compositeLayers', 'rendering', ''],
|
||||
['paintCount', 'rendering', ''],
|
||||
['extHostHeapUsedBefore', 'extHost', 'MB'],
|
||||
['extHostHeapUsedAfter', 'extHost', 'MB'],
|
||||
['extHostHeapDelta', 'extHost', 'MB'],
|
||||
['extHostHeapDeltaPostGC', 'extHost', 'MB'],
|
||||
];
|
||||
|
||||
module.exports = {
|
||||
ROOT,
|
||||
DATA_DIR,
|
||||
METRIC_DEFS,
|
||||
loadConfig,
|
||||
getElectronPath,
|
||||
getRepoRoot,
|
||||
isVersionString,
|
||||
resolveBuild,
|
||||
preseedStorage,
|
||||
buildEnv,
|
||||
buildArgs,
|
||||
writeSettings,
|
||||
prepareRunDir,
|
||||
median,
|
||||
removeOutliers,
|
||||
robustStats,
|
||||
welchTTest,
|
||||
linearRegressionSlope,
|
||||
summarize,
|
||||
markDuration,
|
||||
launchVSCode,
|
||||
getNextExtHostInspectPort,
|
||||
connectToExtHostInspector,
|
||||
};
|
||||
@@ -0,0 +1,37 @@
|
||||
{
|
||||
"perfRegression": {
|
||||
// VS Code version, "insiders", or a commit hash (7-40 hex chars)
|
||||
"baselineBuild": "1.116.0",
|
||||
|
||||
// Number of benchmark iterations per scenario
|
||||
"runsPerScenario": 5,
|
||||
|
||||
// Default fraction above baseline that triggers a regression (0.2 = 20%).
|
||||
// Per-metric overrides below take precedence when set.
|
||||
"regressionThreshold": 0.2,
|
||||
|
||||
// Per-metric regression thresholds.
|
||||
// - A plain number (0-1) is a fraction, e.g. 0.2 = 20% above baseline.
|
||||
// - A string ending in the metric's unit (e.g. "100ms") is an absolute delta.
|
||||
// Metrics not listed here use regressionThreshold above.
|
||||
"metricThresholds": {
|
||||
"timeToFirstToken": "100ms",
|
||||
"timeToComplete": 0.2,
|
||||
"layoutCount": 0.2,
|
||||
"recalcStyleCount": 0.2,
|
||||
"forcedReflowCount": 0.2,
|
||||
"longTaskCount": 0.2,
|
||||
"longAnimationFrameCount": 0.2
|
||||
}
|
||||
},
|
||||
"memLeaks": {
|
||||
// Number of open→work→reset cycles
|
||||
"iterations": 3,
|
||||
|
||||
// Max acceptable total residual heap growth in MB.
|
||||
// Each iteration cycles through ALL scenarios (text, code blocks,
|
||||
// tool calls, thinking, terminal, multi-turn, etc.), so this needs
|
||||
// to account for V8 internal caches that aren't immediately reclaimed.
|
||||
"leakThresholdMB": 10
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
// perf-benchmark-marker
|
||||
|
||||
/**
|
||||
* Fixture for chat-simulation benchmarks.
|
||||
* Simplified from src/vs/base/common/arrays.ts for stable perf testing.
|
||||
*/
|
||||
|
||||
export function coalesce<T>(array: ReadonlyArray<T | undefined | null>): T[] {
|
||||
return array.filter((e): e is T => e !== undefined && e !== null);
|
||||
}
|
||||
|
||||
export function groupBy<T>(data: ReadonlyArray<T>, groupFn: (element: T) => string): { [key: string]: T[] } {
|
||||
const result: { [key: string]: T[] } = {};
|
||||
for (const element of data) {
|
||||
const key = groupFn(element);
|
||||
(result[key] ??= []).push(element);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
export function distinct<T>(array: ReadonlyArray<T>, keyFn: (t: T) => any = t => t): T[] {
|
||||
const seen = new Set<any>();
|
||||
return array.filter(element => {
|
||||
const key = keyFn(element);
|
||||
if (seen.has(key)) { return false; }
|
||||
seen.add(key);
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
export function firstOrDefault<T>(array: ReadonlyArray<T>): T | undefined;
|
||||
export function firstOrDefault<T>(array: ReadonlyArray<T>, defaultValue: T): T;
|
||||
export function firstOrDefault<T>(array: ReadonlyArray<T>, defaultValue?: T): T | undefined {
|
||||
return array.length > 0 ? array[0] : defaultValue;
|
||||
}
|
||||
|
||||
export function lastOrDefault<T>(array: ReadonlyArray<T>): T | undefined;
|
||||
export function lastOrDefault<T>(array: ReadonlyArray<T>, defaultValue: T): T;
|
||||
export function lastOrDefault<T>(array: ReadonlyArray<T>, defaultValue?: T): T | undefined {
|
||||
return array.length > 0 ? array[array.length - 1] : defaultValue;
|
||||
}
|
||||
|
||||
export function binarySearch<T>(array: ReadonlyArray<T>, key: T, comparator: (a: T, b: T) => number): number {
|
||||
let low = 0;
|
||||
let high = array.length - 1;
|
||||
while (low <= high) {
|
||||
const mid = ((low + high) / 2) | 0;
|
||||
const comp = comparator(array[mid], key);
|
||||
if (comp < 0) { low = mid + 1; }
|
||||
else if (comp > 0) { high = mid - 1; }
|
||||
else { return mid; }
|
||||
}
|
||||
return -(low + 1);
|
||||
}
|
||||
|
||||
export function insertSorted<T>(array: T[], element: T, comparator: (a: T, b: T) => number): void {
|
||||
const idx = binarySearch(array, element, comparator);
|
||||
const insertIdx = idx < 0 ? ~idx : idx;
|
||||
array.splice(insertIdx, 0, element);
|
||||
}
|
||||
|
||||
export function flatten<T>(arr: T[][]): T[] {
|
||||
return ([] as T[]).concat(...arr);
|
||||
}
|
||||
|
||||
export function range(to: number): number[];
|
||||
export function range(from: number, to: number): number[];
|
||||
export function range(arg: number, to?: number): number[] {
|
||||
const from = to !== undefined ? arg : 0;
|
||||
const end = to !== undefined ? to : arg;
|
||||
const result: number[] = [];
|
||||
for (let i = from; i < end; i++) { result.push(i); }
|
||||
return result;
|
||||
}
|
||||
|
||||
export function tail<T>(array: T[]): [T[], T] {
|
||||
if (array.length === 0) { throw new Error('Invalid tail call'); }
|
||||
return [array.slice(0, array.length - 1), array[array.length - 1]];
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
// perf-benchmark-marker
|
||||
|
||||
/**
|
||||
* Fixture for chat-simulation benchmarks.
|
||||
* Simplified from src/vs/base/common/async.ts for stable perf testing.
|
||||
*/
|
||||
|
||||
import { IDisposable } from './lifecycle';
|
||||
import { CancellationError } from './errors';
|
||||
|
||||
export class Throttler {
|
||||
private activePromise: Promise<any> | null = null;
|
||||
private queuedPromiseFactory: (() => Promise<any>) | null = null;
|
||||
|
||||
queue<T>(promiseFactory: () => Promise<T>): Promise<T> {
|
||||
if (this.activePromise) {
|
||||
this.queuedPromiseFactory = promiseFactory;
|
||||
return this.activePromise as Promise<T>;
|
||||
}
|
||||
this.activePromise = promiseFactory();
|
||||
return this.activePromise.finally(() => {
|
||||
this.activePromise = null;
|
||||
if (this.queuedPromiseFactory) {
|
||||
const factory = this.queuedPromiseFactory;
|
||||
this.queuedPromiseFactory = null;
|
||||
return this.queue(factory);
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
export class Delayer<T> implements IDisposable {
|
||||
private timeout: any;
|
||||
private task: (() => T | Promise<T>) | null = null;
|
||||
|
||||
constructor(public defaultDelay: number) { }
|
||||
|
||||
trigger(task: () => T | Promise<T>, delay: number = this.defaultDelay): Promise<T> {
|
||||
this.task = task;
|
||||
this.cancelTimeout();
|
||||
return new Promise<T>((resolve, reject) => {
|
||||
this.timeout = setTimeout(() => {
|
||||
this.timeout = null;
|
||||
try { resolve(this.task!()); } catch (e) { reject(e); }
|
||||
this.task = null;
|
||||
}, delay);
|
||||
});
|
||||
}
|
||||
|
||||
private cancelTimeout(): void {
|
||||
if (this.timeout !== null) {
|
||||
clearTimeout(this.timeout);
|
||||
this.timeout = null;
|
||||
}
|
||||
}
|
||||
|
||||
dispose(): void {
|
||||
this.cancelTimeout();
|
||||
}
|
||||
}
|
||||
|
||||
export class RunOnceScheduler implements IDisposable {
|
||||
private runner: (() => void) | null;
|
||||
private timeout: any;
|
||||
|
||||
constructor(runner: () => void, private delay: number) {
|
||||
this.runner = runner;
|
||||
}
|
||||
|
||||
schedule(delay = this.delay): void {
|
||||
this.cancel();
|
||||
this.timeout = setTimeout(() => {
|
||||
this.timeout = null;
|
||||
this.runner?.();
|
||||
}, delay);
|
||||
}
|
||||
|
||||
cancel(): void {
|
||||
if (this.timeout !== null) {
|
||||
clearTimeout(this.timeout);
|
||||
this.timeout = null;
|
||||
}
|
||||
}
|
||||
|
||||
isScheduled(): boolean { return this.timeout !== null; }
|
||||
|
||||
dispose(): void {
|
||||
this.cancel();
|
||||
this.runner = null;
|
||||
}
|
||||
}
|
||||
|
||||
export class Queue<T> {
|
||||
private readonly queue: Array<() => Promise<T>> = [];
|
||||
private running = false;
|
||||
|
||||
async enqueue(factory: () => Promise<T>): Promise<T> {
|
||||
return new Promise<T>((resolve, reject) => {
|
||||
this.queue.push(() => factory().then(resolve, reject));
|
||||
if (!this.running) { this.processQueue(); }
|
||||
});
|
||||
}
|
||||
|
||||
private async processQueue(): Promise<void> {
|
||||
this.running = true;
|
||||
while (this.queue.length > 0) {
|
||||
const task = this.queue.shift()!;
|
||||
await task();
|
||||
}
|
||||
this.running = false;
|
||||
}
|
||||
|
||||
get size(): number { return this.queue.length; }
|
||||
}
|
||||
|
||||
export function timeout(millis: number): Promise<void> {
|
||||
return new Promise<void>(resolve => setTimeout(resolve, millis));
|
||||
}
|
||||
|
||||
export async function retry<T>(task: () => Promise<T>, delay: number, retries: number): Promise<T> {
|
||||
let lastError: Error | undefined;
|
||||
for (let i = 0; i < retries; i++) {
|
||||
try { return await task(); }
|
||||
catch (error) { lastError = error as Error; await timeout(delay); }
|
||||
}
|
||||
throw lastError;
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
// perf-benchmark-marker
|
||||
|
||||
/**
|
||||
* Fixture for chat-simulation benchmarks.
|
||||
* Simplified from src/vs/base/common/errors.ts for stable perf testing.
|
||||
*/
|
||||
|
||||
export interface ErrorListenerCallback {
|
||||
(error: any): void;
|
||||
}
|
||||
|
||||
export interface ErrorListenerUnbind {
|
||||
(): void;
|
||||
}
|
||||
|
||||
const _errorListeners: ErrorListenerCallback[] = [];
|
||||
|
||||
export function setUnexpectedErrorHandler(handler: ErrorListenerCallback): void {
|
||||
_errorListeners.length = 0;
|
||||
_errorListeners.push(handler);
|
||||
}
|
||||
|
||||
export function onUnexpectedError(e: any): void {
|
||||
if (!isCancellationError(e)) {
|
||||
for (const listener of _errorListeners) {
|
||||
try { listener(e); } catch { }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function onUnexpectedExternalError(e: any): void {
|
||||
if (!isCancellationError(e)) {
|
||||
for (const listener of _errorListeners) {
|
||||
try { listener(e); } catch { }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function transformErrorForSerialization(error: any): any {
|
||||
if (error instanceof Error) {
|
||||
const { name, message, stack } = error;
|
||||
return { $isError: true, name, message, stack };
|
||||
}
|
||||
return error;
|
||||
}
|
||||
|
||||
const canceledName = 'Canceled';
|
||||
|
||||
export function isCancellationError(error: any): boolean {
|
||||
if (error instanceof CancellationError) { return true; }
|
||||
return error instanceof Error && error.name === canceledName && error.message === canceledName;
|
||||
}
|
||||
|
||||
export class CancellationError extends Error {
|
||||
constructor() {
|
||||
super(canceledName);
|
||||
this.name = this.message;
|
||||
}
|
||||
}
|
||||
|
||||
export class NotSupportedError extends Error {
|
||||
constructor(message?: string) {
|
||||
super(message || 'NotSupported');
|
||||
}
|
||||
}
|
||||
|
||||
export class NotImplementedError extends Error {
|
||||
constructor(message?: string) {
|
||||
super(message || 'NotImplemented');
|
||||
}
|
||||
}
|
||||
|
||||
export class IllegalArgumentError extends Error {
|
||||
constructor(message?: string) {
|
||||
super(message || 'Illegal argument');
|
||||
}
|
||||
}
|
||||
|
||||
export class BugIndicatingError extends Error {
|
||||
constructor(message?: string) {
|
||||
super(message || 'Bug Indicating Error');
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
// perf-benchmark-marker
|
||||
|
||||
/**
|
||||
* Fixture for chat-simulation benchmarks.
|
||||
* Simplified from src/vs/base/common/event.ts for stable perf testing.
|
||||
*/
|
||||
|
||||
import { IDisposable, DisposableStore } from './lifecycle';
|
||||
|
||||
export interface Event<T> {
|
||||
(listener: (e: T) => any, thisArgs?: any, disposables?: IDisposable[]): IDisposable;
|
||||
}
|
||||
|
||||
export namespace Event {
|
||||
export const None: Event<any> = () => ({ dispose() { } });
|
||||
|
||||
export function once<T>(event: Event<T>): Event<T> {
|
||||
return (listener, thisArgs?, disposables?) => {
|
||||
let didFire = false;
|
||||
const result = event(e => {
|
||||
if (didFire) { return; }
|
||||
didFire = true;
|
||||
return listener.call(thisArgs, e);
|
||||
}, null, disposables);
|
||||
if (didFire) { result.dispose(); }
|
||||
return result;
|
||||
};
|
||||
}
|
||||
|
||||
export function map<I, O>(event: Event<I>, map: (i: I) => O): Event<O> {
|
||||
return (listener, thisArgs?, disposables?) =>
|
||||
event(i => listener.call(thisArgs, map(i)), null, disposables);
|
||||
}
|
||||
|
||||
export function filter<T>(event: Event<T>, filter: (e: T) => boolean): Event<T> {
|
||||
return (listener, thisArgs?, disposables?) =>
|
||||
event(e => filter(e) && listener.call(thisArgs, e), null, disposables);
|
||||
}
|
||||
|
||||
export function debounce<T>(event: Event<T>, merge: (last: T | undefined, e: T) => T, delay: number = 100): Event<T> {
|
||||
let subscription: IDisposable;
|
||||
let output: T | undefined;
|
||||
let handle: any;
|
||||
return (listener, thisArgs?, disposables?) => {
|
||||
subscription = event(cur => {
|
||||
output = merge(output, cur);
|
||||
clearTimeout(handle);
|
||||
handle = setTimeout(() => {
|
||||
const e = output!;
|
||||
output = undefined;
|
||||
listener.call(thisArgs, e);
|
||||
}, delay);
|
||||
});
|
||||
return { dispose() { subscription.dispose(); clearTimeout(handle); } };
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
export class Emitter<T> {
|
||||
private readonly _listeners = new Set<(e: T) => void>();
|
||||
private _disposed = false;
|
||||
|
||||
readonly event: Event<T> = (listener: (e: T) => void) => {
|
||||
if (this._disposed) { return { dispose() { } }; }
|
||||
this._listeners.add(listener);
|
||||
return {
|
||||
dispose: () => { this._listeners.delete(listener); }
|
||||
};
|
||||
};
|
||||
|
||||
fire(event: T): void {
|
||||
if (this._disposed) { return; }
|
||||
for (const listener of [...this._listeners]) {
|
||||
try { listener(event); } catch { }
|
||||
}
|
||||
}
|
||||
|
||||
dispose(): void {
|
||||
if (this._disposed) { return; }
|
||||
this._disposed = true;
|
||||
this._listeners.clear();
|
||||
}
|
||||
|
||||
get hasListeners(): boolean { return this._listeners.size > 0; }
|
||||
}
|
||||
|
||||
export class PauseableEmitter<T> extends Emitter<T> {
|
||||
private _isPaused = false;
|
||||
private _queue: T[] = [];
|
||||
|
||||
pause(): void { this._isPaused = true; }
|
||||
|
||||
resume(): void {
|
||||
this._isPaused = false;
|
||||
while (this._queue.length > 0) {
|
||||
super.fire(this._queue.shift()!);
|
||||
}
|
||||
}
|
||||
|
||||
override fire(event: T): void {
|
||||
if (this._isPaused) { this._queue.push(event); }
|
||||
else { super.fire(event); }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
// perf-benchmark-marker
|
||||
|
||||
/**
|
||||
* Fixture for chat-simulation benchmarks.
|
||||
* Simplified from src/vs/base/common/lifecycle.ts for stable perf testing.
|
||||
*/
|
||||
|
||||
export interface IDisposable {
|
||||
dispose(): void;
|
||||
}
|
||||
|
||||
export function isDisposable<T extends object>(thing: T): thing is T & IDisposable {
|
||||
return typeof (thing as IDisposable).dispose === 'function'
|
||||
&& (thing as IDisposable).dispose.length === 0;
|
||||
}
|
||||
|
||||
export function dispose<T extends IDisposable>(disposable: T): T;
|
||||
export function dispose<T extends IDisposable>(disposable: T | undefined): T | undefined;
|
||||
export function dispose<T extends IDisposable>(disposables: T[]): T[];
|
||||
export function dispose<T extends IDisposable>(disposables: readonly T[]): readonly T[];
|
||||
export function dispose<T extends IDisposable>(arg: T | T[] | undefined): any {
|
||||
if (Array.isArray(arg)) {
|
||||
const errors: any[] = [];
|
||||
for (const d of arg) {
|
||||
try { d.dispose(); } catch (e) { errors.push(e); }
|
||||
}
|
||||
if (errors.length > 0) { throw new Error(`Dispose errors: ${errors.length}`); }
|
||||
return arg;
|
||||
} else if (arg) {
|
||||
arg.dispose();
|
||||
return arg;
|
||||
}
|
||||
}
|
||||
|
||||
export class DisposableStore implements IDisposable {
|
||||
private readonly _toDispose = new Set<IDisposable>();
|
||||
private _isDisposed = false;
|
||||
|
||||
dispose(): void {
|
||||
if (this._isDisposed) { return; }
|
||||
this._isDisposed = true;
|
||||
this.clear();
|
||||
}
|
||||
|
||||
clear(): void {
|
||||
if (this._toDispose.size === 0) { return; }
|
||||
const iter = this._toDispose.values();
|
||||
this._toDispose.clear();
|
||||
for (const item of iter) {
|
||||
try { item.dispose(); } catch { }
|
||||
}
|
||||
}
|
||||
|
||||
add<T extends IDisposable>(o: T): T {
|
||||
if (this._isDisposed) {
|
||||
console.warn('Adding to a disposed DisposableStore');
|
||||
return o;
|
||||
}
|
||||
this._toDispose.add(o);
|
||||
return o;
|
||||
}
|
||||
|
||||
get size(): number { return this._toDispose.size; }
|
||||
}
|
||||
|
||||
export abstract class Disposable implements IDisposable {
|
||||
private readonly _store = new DisposableStore();
|
||||
|
||||
dispose(): void {
|
||||
this._store.dispose();
|
||||
}
|
||||
|
||||
protected _register<T extends IDisposable>(o: T): T {
|
||||
return this._store.add(o);
|
||||
}
|
||||
}
|
||||
|
||||
export class MutableDisposable<T extends IDisposable> implements IDisposable {
|
||||
private _value?: T;
|
||||
private _isDisposed = false;
|
||||
|
||||
get value(): T | undefined { return this._isDisposed ? undefined : this._value; }
|
||||
|
||||
set value(value: T | undefined) {
|
||||
if (this._isDisposed || value === this._value) { return; }
|
||||
this._value?.dispose();
|
||||
this._value = value;
|
||||
}
|
||||
|
||||
dispose(): void {
|
||||
this._isDisposed = true;
|
||||
this._value?.dispose();
|
||||
this._value = undefined;
|
||||
}
|
||||
}
|
||||
|
||||
export class DisposableMap<K, V extends IDisposable> implements IDisposable {
|
||||
private readonly _map = new Map<K, V>();
|
||||
private _isDisposed = false;
|
||||
|
||||
set(key: K, value: V): void {
|
||||
const existing = this._map.get(key);
|
||||
if (existing !== value) {
|
||||
existing?.dispose();
|
||||
this._map.set(key, value);
|
||||
}
|
||||
}
|
||||
|
||||
get(key: K): V | undefined { return this._map.get(key); }
|
||||
|
||||
delete(key: K): void {
|
||||
this._map.get(key)?.dispose();
|
||||
this._map.delete(key);
|
||||
}
|
||||
|
||||
dispose(): void {
|
||||
if (this._isDisposed) { return; }
|
||||
this._isDisposed = true;
|
||||
for (const [, v] of this._map) { v.dispose(); }
|
||||
this._map.clear();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
// perf-benchmark-marker
|
||||
|
||||
/**
|
||||
* Fixture for chat-simulation benchmarks.
|
||||
* Simplified from src/vs/base/common/strings.ts for stable perf testing.
|
||||
*/
|
||||
|
||||
export function format(value: string, ...args: any[]): string {
|
||||
return value.replace(/{(\d+)}/g, (match, index) => {
|
||||
const i = parseInt(index, 10);
|
||||
return i >= 0 && i < args.length ? `${args[i]}` : match;
|
||||
});
|
||||
}
|
||||
|
||||
export function escape(value: string): string {
|
||||
return value.replace(/[<>&"']/g, ch => {
|
||||
switch (ch) {
|
||||
case '<': return '<';
|
||||
case '>': return '>';
|
||||
case '&': return '&';
|
||||
case '"': return '"';
|
||||
case '\'': return ''';
|
||||
default: return ch;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
export function trim(value: string, ch: string = ' '): string {
|
||||
let start = 0;
|
||||
let end = value.length;
|
||||
while (start < end && value[start] === ch) { start++; }
|
||||
while (end > start && value[end - 1] === ch) { end--; }
|
||||
return value.substring(start, end);
|
||||
}
|
||||
|
||||
export function equalsIgnoreCase(a: string, b: string): boolean {
|
||||
return a.length === b.length && a.toLowerCase() === b.toLowerCase();
|
||||
}
|
||||
|
||||
export function startsWithIgnoreCase(str: string, candidate: string): boolean {
|
||||
if (str.length < candidate.length) { return false; }
|
||||
return str.substring(0, candidate.length).toLowerCase() === candidate.toLowerCase();
|
||||
}
|
||||
|
||||
export function commonPrefixLength(a: string, b: string): number {
|
||||
const len = Math.min(a.length, b.length);
|
||||
for (let i = 0; i < len; i++) {
|
||||
if (a.charCodeAt(i) !== b.charCodeAt(i)) { return i; }
|
||||
}
|
||||
return len;
|
||||
}
|
||||
|
||||
export function commonSuffixLength(a: string, b: string): number {
|
||||
const len = Math.min(a.length, b.length);
|
||||
for (let i = 0; i < len; i++) {
|
||||
if (a.charCodeAt(a.length - 1 - i) !== b.charCodeAt(b.length - 1 - i)) { return i; }
|
||||
}
|
||||
return len;
|
||||
}
|
||||
|
||||
export function splitLines(str: string): string[] {
|
||||
return str.split(/\r\n|\r|\n/);
|
||||
}
|
||||
|
||||
export function regExpLeadsToEndlessLoop(regexp: RegExp): boolean {
|
||||
if (regexp.source === '^' || regexp.source === '^$' || regexp.source === '$') {
|
||||
return false;
|
||||
}
|
||||
return !regexp.exec('')?.length;
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
// perf-benchmark-marker
|
||||
|
||||
/**
|
||||
* Fixture for chat-simulation benchmarks.
|
||||
* Simplified from src/vs/base/common/types.ts for stable perf testing.
|
||||
*/
|
||||
|
||||
export function isString(thing: unknown): thing is string {
|
||||
return typeof thing === 'string';
|
||||
}
|
||||
|
||||
export function isNumber(thing: unknown): thing is number {
|
||||
return typeof thing === 'number' && !isNaN(thing);
|
||||
}
|
||||
|
||||
export function isBoolean(thing: unknown): thing is boolean {
|
||||
return thing === true || thing === false;
|
||||
}
|
||||
|
||||
export function isUndefined(thing: unknown): thing is undefined {
|
||||
return typeof thing === 'undefined';
|
||||
}
|
||||
|
||||
export function isDefined<T>(thing: T | undefined | null): thing is T {
|
||||
return !isUndefinedOrNull(thing);
|
||||
}
|
||||
|
||||
export function isUndefinedOrNull(thing: unknown): thing is undefined | null {
|
||||
return isUndefined(thing) || thing === null;
|
||||
}
|
||||
|
||||
export function isFunction(thing: unknown): thing is Function {
|
||||
return typeof thing === 'function';
|
||||
}
|
||||
|
||||
export function isObject(thing: unknown): thing is object {
|
||||
return typeof thing === 'object'
|
||||
&& thing !== null
|
||||
&& !Array.isArray(thing)
|
||||
&& !(thing instanceof RegExp)
|
||||
&& !(thing instanceof Date);
|
||||
}
|
||||
|
||||
export function isArray(thing: unknown): thing is unknown[] {
|
||||
return Array.isArray(thing);
|
||||
}
|
||||
|
||||
export function assertType(condition: unknown, type?: string): asserts condition {
|
||||
if (!condition) {
|
||||
throw new Error(type ? `Unexpected type, expected '${type}'` : 'Unexpected type');
|
||||
}
|
||||
}
|
||||
|
||||
export function assertIsDefined<T>(thing: T | undefined | null): T {
|
||||
if (isUndefinedOrNull(thing)) {
|
||||
throw new Error('Assertion failed: value is undefined or null');
|
||||
}
|
||||
return thing;
|
||||
}
|
||||
|
||||
export function assertAllDefined<T1, T2>(t1: T1 | undefined | null, t2: T2 | undefined | null): [T1, T2] {
|
||||
return [assertIsDefined(t1), assertIsDefined(t2)];
|
||||
}
|
||||
|
||||
export type TypeConstraint = string | Function;
|
||||
|
||||
export function validateConstraints(args: unknown[], constraints: Array<TypeConstraint | undefined>): void {
|
||||
const len = Math.min(args.length, constraints.length);
|
||||
for (let i = 0; i < len; i++) {
|
||||
validateConstraint(args[i], constraints[i]);
|
||||
}
|
||||
}
|
||||
|
||||
export function validateConstraint(arg: unknown, constraint: TypeConstraint | undefined): void {
|
||||
if (isString(constraint)) {
|
||||
if (typeof arg !== constraint) {
|
||||
throw new Error(`argument does not match constraint: typeof ${constraint}`);
|
||||
}
|
||||
} else if (isFunction(constraint)) {
|
||||
try {
|
||||
if (arg instanceof constraint) { return; }
|
||||
} catch { }
|
||||
if (!isUndefinedOrNull(arg) && (arg as any).constructor === constraint) { return; }
|
||||
if (constraint.length === 1 && constraint.call(undefined, arg) === true) { return; }
|
||||
throw new Error('argument does not match one of these constraints: arg instanceof constraint, arg.constructor === constraint, nor constraint(arg) === true');
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
// perf-benchmark-marker
|
||||
|
||||
/**
|
||||
* Fixture for chat-simulation benchmarks.
|
||||
* Simplified from src/vs/base/common/uri.ts for stable perf testing.
|
||||
*/
|
||||
|
||||
const _empty = '';
|
||||
const _slash = '/';
|
||||
|
||||
export class URI {
|
||||
readonly scheme: string;
|
||||
readonly authority: string;
|
||||
readonly path: string;
|
||||
readonly query: string;
|
||||
readonly fragment: string;
|
||||
|
||||
private constructor(scheme: string, authority: string, path: string, query: string, fragment: string) {
|
||||
this.scheme = scheme;
|
||||
this.authority = authority || _empty;
|
||||
this.path = path || _empty;
|
||||
this.query = query || _empty;
|
||||
this.fragment = fragment || _empty;
|
||||
}
|
||||
|
||||
static file(path: string): URI {
|
||||
let authority = _empty;
|
||||
if (path.length >= 2 && path.charCodeAt(0) === 47 /* / */ && path.charCodeAt(1) === 47 /* / */) {
|
||||
const idx = path.indexOf(_slash, 2);
|
||||
if (idx === -1) {
|
||||
authority = path.substring(2);
|
||||
path = _slash;
|
||||
} else {
|
||||
authority = path.substring(2, idx);
|
||||
path = path.substring(idx) || _slash;
|
||||
}
|
||||
}
|
||||
return new URI('file', authority, path, _empty, _empty);
|
||||
}
|
||||
|
||||
static parse(value: string): URI {
|
||||
const match = /^([a-zA-Z][a-zA-Z0-9+.-]*):\/\/([^/?#]*)([^?#]*)(\?[^#]*)?(#.*)?$/.exec(value);
|
||||
if (!match) { return new URI(_empty, _empty, _empty, _empty, _empty); }
|
||||
return new URI(match[1], match[2], match[3], match[4]?.substring(1) || _empty, match[5]?.substring(1) || _empty);
|
||||
}
|
||||
|
||||
with(change: { scheme?: string; authority?: string; path?: string; query?: string; fragment?: string }): URI {
|
||||
return new URI(
|
||||
change.scheme ?? this.scheme,
|
||||
change.authority ?? this.authority,
|
||||
change.path ?? this.path,
|
||||
change.query ?? this.query,
|
||||
change.fragment ?? this.fragment,
|
||||
);
|
||||
}
|
||||
|
||||
toString(): string {
|
||||
let result = '';
|
||||
if (this.scheme) { result += this.scheme + '://'; }
|
||||
if (this.authority) { result += this.authority; }
|
||||
if (this.path) { result += this.path; }
|
||||
if (this.query) { result += '?' + this.query; }
|
||||
if (this.fragment) { result += '#' + this.fragment; }
|
||||
return result;
|
||||
}
|
||||
|
||||
get fsPath(): string {
|
||||
return this.path;
|
||||
}
|
||||
|
||||
toJSON(): object {
|
||||
return {
|
||||
scheme: this.scheme,
|
||||
authority: this.authority,
|
||||
path: this.path,
|
||||
query: this.query,
|
||||
fragment: this.fragment,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,535 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
// @ts-check
|
||||
|
||||
/**
|
||||
* Merge per-group perf results into a single unified CI summary.
|
||||
*
|
||||
* Called by the CI report job after all matrix groups have finished.
|
||||
* Reads results.json and baseline-*.json from each group directory,
|
||||
* merges all scenarios into one combined report, and writes a single
|
||||
* ci-summary.md file.
|
||||
*
|
||||
* Usage:
|
||||
* node scripts/chat-simulation/merge-ci-summary.js \
|
||||
* --results-dir perf-results \
|
||||
* --output ci-summary.md \
|
||||
* [--leak-summary leak-results/.chat-simulation-data/ci-summary-leak.md] \
|
||||
* [--threshold 0.2]
|
||||
*/
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { welchTTest, loadConfig } = require('./common/utils');
|
||||
|
||||
// -- CLI args ----------------------------------------------------------------
|
||||
|
||||
function parseArgs() {
|
||||
const args = process.argv.slice(2);
|
||||
const opts = {
|
||||
resultsDir: '',
|
||||
output: '',
|
||||
/** @type {string | undefined} */
|
||||
leakSummary: undefined,
|
||||
threshold: 0.2,
|
||||
/** @type {Record<string, number | string>} */
|
||||
metricThresholds: {},
|
||||
};
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
switch (args[i]) {
|
||||
case '--results-dir': opts.resultsDir = args[++i]; break;
|
||||
case '--output': opts.output = args[++i]; break;
|
||||
case '--leak-summary': opts.leakSummary = args[++i]; break;
|
||||
case '--threshold': opts.threshold = parseFloat(args[++i]); break;
|
||||
case '--help': case '-h':
|
||||
console.log([
|
||||
'Merge per-group perf results into a single CI summary.',
|
||||
'',
|
||||
'Options:',
|
||||
' --results-dir <dir> Directory containing perf-results-* subdirs',
|
||||
' --output <path> Output path for ci-summary.md',
|
||||
' --leak-summary <path> Path to ci-summary-leak.md (optional)',
|
||||
' --threshold <frac> Regression threshold fraction (default: 0.2)',
|
||||
].join('\n'));
|
||||
process.exit(0);
|
||||
}
|
||||
}
|
||||
if (!opts.resultsDir || !opts.output) {
|
||||
console.error('Required: --results-dir and --output');
|
||||
process.exit(1);
|
||||
}
|
||||
return opts;
|
||||
}
|
||||
|
||||
// -- Merge logic -------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Find all results.json and baseline-*.json files across group directories,
|
||||
* merge scenarios into a single combined report.
|
||||
* @param {string} resultsDir
|
||||
*/
|
||||
function mergeResults(resultsDir) {
|
||||
const groupDirs = fs.readdirSync(resultsDir)
|
||||
.filter(d => d.startsWith('perf-results-'))
|
||||
.map(d => path.join(resultsDir, d))
|
||||
.filter(d => fs.statSync(d).isDirectory());
|
||||
|
||||
if (groupDirs.length === 0) {
|
||||
console.error(`No perf-results-* directories found in ${resultsDir}`);
|
||||
return null;
|
||||
}
|
||||
|
||||
/** @type {Record<string, any>} */
|
||||
const mergedScenarios = {};
|
||||
/** @type {Record<string, any>} */
|
||||
const mergedBaselineScenarios = {};
|
||||
let runsPerScenario = 0;
|
||||
let platform = 'linux';
|
||||
/** @type {string | undefined} */
|
||||
let baselineBuildVersion;
|
||||
/** @type {string | undefined} */
|
||||
let threshold;
|
||||
|
||||
// Read per-metric thresholds from config.jsonc (same source as the perf script)
|
||||
const perfConfig = loadConfig('perfRegression');
|
||||
/** @type {Record<string, number | string>} */
|
||||
const metricThresholds = perfConfig.metricThresholds ?? {};
|
||||
|
||||
for (const groupDir of groupDirs) {
|
||||
// Find results.json (may be in a timestamped subdir under .chat-simulation-data)
|
||||
const simDataDir = path.join(groupDir, '.chat-simulation-data');
|
||||
if (!fs.existsSync(simDataDir)) { continue; }
|
||||
|
||||
// Search for results.json in timestamped subdirs
|
||||
const subdirs = fs.readdirSync(simDataDir).filter(d => {
|
||||
const full = path.join(simDataDir, d);
|
||||
return fs.statSync(full).isDirectory() && /^\d{4}-/.test(d);
|
||||
});
|
||||
|
||||
for (const subdir of subdirs) {
|
||||
const resultsPath = path.join(simDataDir, subdir, 'results.json');
|
||||
if (fs.existsSync(resultsPath)) {
|
||||
const results = JSON.parse(fs.readFileSync(resultsPath, 'utf-8'));
|
||||
runsPerScenario = results.runsPerScenario || runsPerScenario;
|
||||
platform = results.platform || platform;
|
||||
for (const [scenario, data] of Object.entries(results.scenarios || {})) {
|
||||
mergedScenarios[scenario] = data;
|
||||
}
|
||||
}
|
||||
|
||||
// Find baseline-*.json in the same dir
|
||||
const baselineFiles = fs.readdirSync(path.join(simDataDir, subdir))
|
||||
.filter(f => f.startsWith('baseline-') && f.endsWith('.json'));
|
||||
for (const bf of baselineFiles) {
|
||||
const baseline = JSON.parse(fs.readFileSync(path.join(simDataDir, subdir, bf), 'utf-8'));
|
||||
baselineBuildVersion = baseline.baselineBuildVersion || baselineBuildVersion;
|
||||
for (const [scenario, data] of Object.entries(baseline.scenarios || {})) {
|
||||
mergedBaselineScenarios[scenario] = data;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Also check for baseline cached at top-level .chat-simulation-data
|
||||
const topBaselines = fs.readdirSync(simDataDir)
|
||||
.filter(f => f.startsWith('baseline-') && f.endsWith('.json'));
|
||||
for (const bf of topBaselines) {
|
||||
const baseline = JSON.parse(fs.readFileSync(path.join(simDataDir, bf), 'utf-8'));
|
||||
baselineBuildVersion = baseline.baselineBuildVersion || baselineBuildVersion;
|
||||
for (const [scenario, data] of Object.entries(baseline.scenarios || {})) {
|
||||
mergedBaselineScenarios[scenario] = data;
|
||||
}
|
||||
}
|
||||
|
||||
// Read threshold/metricThresholds from the group's ci-summary or config
|
||||
const ciSummaryPath = path.join(simDataDir, 'ci-summary.md');
|
||||
if (fs.existsSync(ciSummaryPath)) {
|
||||
const content = fs.readFileSync(ciSummaryPath, 'utf-8');
|
||||
const thresholdMatch = content.match(/Regression threshold\*\* \| (\d+)%/);
|
||||
if (thresholdMatch) {
|
||||
threshold = thresholdMatch[1];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const mergedReport = {
|
||||
timestamp: new Date().toISOString(),
|
||||
platform,
|
||||
runsPerScenario,
|
||||
scenarios: mergedScenarios,
|
||||
};
|
||||
|
||||
const mergedBaseline = Object.keys(mergedBaselineScenarios).length > 0
|
||||
? { baselineBuildVersion, scenarios: mergedBaselineScenarios }
|
||||
: null;
|
||||
|
||||
return { report: mergedReport, baseline: mergedBaseline, baselineBuildVersion, threshold: threshold ? parseInt(threshold, 10) / 100 : undefined, metricThresholds };
|
||||
}
|
||||
|
||||
// -- Summary generation (unified, single-header format) ----------------------
|
||||
|
||||
const GITHUB_REPO = 'https://github.com/microsoft/vscode';
|
||||
|
||||
/** @param {string} label */
|
||||
function formatBuildLink(label) {
|
||||
if (/^[0-9a-f]{7,40}$/.test(label)) {
|
||||
return `[\`${label.substring(0, 7)}\`](${GITHUB_REPO}/commit/${label})`;
|
||||
}
|
||||
if (/^\d+\.\d+\.\d+/.test(label)) {
|
||||
return `[\`${label}\`](${GITHUB_REPO}/releases/tag/${label})`;
|
||||
}
|
||||
return `\`${label}\``;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} base
|
||||
* @param {string} test
|
||||
*/
|
||||
function formatCompareLink(base, test) {
|
||||
const isRef = (/** @type {string} */ v) => /^[0-9a-f]{7,40}$/.test(v) || /^\d+\.\d+\.\d+/.test(v);
|
||||
if (!isRef(base) || !isRef(test)) { return ''; }
|
||||
return `[compare](${GITHUB_REPO}/compare/${base}...${test})`;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {{ type: string, value: number }} threshold
|
||||
* @param {number} change
|
||||
* @param {number} absoluteDelta
|
||||
*/
|
||||
function exceedsThreshold(threshold, change, absoluteDelta) {
|
||||
if (threshold.type === 'absolute') { return absoluteDelta > threshold.value; }
|
||||
return change > threshold.value;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {{ threshold: number, metricThresholds?: Record<string, number | string> }} opts
|
||||
* @param {string} metric
|
||||
*/
|
||||
function getMetricThreshold(opts, metric) {
|
||||
const raw = opts.metricThresholds?.[metric];
|
||||
if (raw !== undefined) {
|
||||
const num = typeof raw === 'number' ? raw : parseFloat(/** @type {string} */(raw));
|
||||
return typeof raw === 'number' ? { type: 'fraction', value: num } : { type: 'absolute', value: num };
|
||||
}
|
||||
return { type: 'fraction', value: opts.threshold };
|
||||
}
|
||||
|
||||
/** @param {number} v */
|
||||
function round2(v) { return Math.round(v * 100) / 100; }
|
||||
|
||||
/**
|
||||
* Generate a unified Markdown summary for all scenarios.
|
||||
*
|
||||
* @param {Record<string, any>} jsonReport
|
||||
* @param {Record<string, any> | null} baseline
|
||||
* @param {{ threshold: number, metricThresholds?: Record<string, number | string>, runs: number, baselineBuild?: string, build?: string }} opts
|
||||
*/
|
||||
function generateUnifiedSummary(jsonReport, baseline, opts) {
|
||||
const baseLabel = opts.baselineBuild || 'baseline';
|
||||
const testLabel = opts.build || 'dev (local)';
|
||||
const baseLink = formatBuildLink(baseLabel);
|
||||
const testLink = formatBuildLink(testLabel);
|
||||
const compareLink = formatCompareLink(baseLabel, testLabel);
|
||||
|
||||
const allMetrics = [
|
||||
['timeToFirstToken', 'timing', 'ms'],
|
||||
['timeToComplete', 'timing', 'ms'],
|
||||
['layoutCount', 'rendering', ''],
|
||||
['recalcStyleCount', 'rendering', ''],
|
||||
['forcedReflowCount', 'rendering', ''],
|
||||
['longTaskCount', 'rendering', ''],
|
||||
['longAnimationFrameCount', 'rendering', ''],
|
||||
['longAnimationFrameTotalMs', 'rendering', 'ms'],
|
||||
['frameCount', 'rendering', ''],
|
||||
['compositeLayers', 'rendering', ''],
|
||||
['paintCount', 'rendering', ''],
|
||||
['heapDelta', 'memory', 'MB'],
|
||||
['heapDeltaPostGC', 'memory', 'MB'],
|
||||
['gcDurationMs', 'memory', 'ms'],
|
||||
['extHostHeapDelta', 'extHost', 'MB'],
|
||||
['extHostHeapDeltaPostGC', 'extHost', 'MB'],
|
||||
];
|
||||
const regressionMetricNames = new Set([
|
||||
'timeToFirstToken', 'timeToComplete', 'layoutCount', 'recalcStyleCount',
|
||||
'forcedReflowCount', 'longTaskCount', 'longAnimationFrameCount',
|
||||
]);
|
||||
|
||||
const lines = [];
|
||||
const scenarios = Object.keys(jsonReport.scenarios);
|
||||
|
||||
// -- Collect verdicts ------------------------------------------------
|
||||
/** @type {Map<string, { metric: string, verdict: string, change: number, pValue: string, basStr: string, curStr: string }[]>} */
|
||||
const scenarioVerdicts = new Map();
|
||||
let totalRegressions = 0;
|
||||
let totalImprovements = 0;
|
||||
|
||||
for (const scenario of scenarios) {
|
||||
const current = jsonReport.scenarios[scenario];
|
||||
const base = baseline?.scenarios?.[scenario];
|
||||
/** @type {{ metric: string, verdict: string, change: number, pValue: string, basStr: string, curStr: string }[]} */
|
||||
const verdicts = [];
|
||||
|
||||
if (base) {
|
||||
for (const [metric, group, unit] of allMetrics) {
|
||||
const cur = current[group]?.[metric];
|
||||
const bas = base[group]?.[metric];
|
||||
if (!cur || !bas || bas.median === null || bas.median === undefined) { continue; }
|
||||
|
||||
const change = bas.median !== 0 ? (cur.median - bas.median) / bas.median : 0;
|
||||
const isRegressionMetric = regressionMetricNames.has(metric);
|
||||
|
||||
const curRaw = (current.rawRuns || []).map((/** @type {any} */ r) => r[metric]).filter((/** @type {any} */ v) => v >= 0);
|
||||
const basRaw = (base.rawRuns || []).map((/** @type {any} */ r) => r[metric]).filter((/** @type {any} */ v) => v >= 0);
|
||||
const ttest = welchTTest(basRaw, curRaw);
|
||||
const pStr = ttest ? `${ttest.pValue}` : 'n/a';
|
||||
|
||||
const metricThreshold = getMetricThreshold(opts, metric);
|
||||
const absoluteDelta = cur.median - bas.median;
|
||||
let verdict = '';
|
||||
if (isRegressionMetric) {
|
||||
if (exceedsThreshold(metricThreshold, change, absoluteDelta)) {
|
||||
if (!ttest || ttest.significant) {
|
||||
verdict = 'REGRESSION';
|
||||
totalRegressions++;
|
||||
} else {
|
||||
verdict = 'noise';
|
||||
}
|
||||
} else if (exceedsThreshold(metricThreshold, -change, -absoluteDelta) && ttest?.significant) {
|
||||
verdict = 'improved';
|
||||
totalImprovements++;
|
||||
} else {
|
||||
verdict = 'ok';
|
||||
}
|
||||
} else {
|
||||
verdict = 'info';
|
||||
}
|
||||
|
||||
const basStr = `${bas.median}${unit} \xb1${bas.stddev}${unit}`;
|
||||
const curStr = `${cur.median}${unit} \xb1${cur.stddev}${unit}`;
|
||||
verdicts.push({ metric, verdict, change, pValue: pStr, basStr, curStr });
|
||||
}
|
||||
}
|
||||
scenarioVerdicts.set(scenario, verdicts);
|
||||
}
|
||||
|
||||
// -- Header ----------------------------------------------------------
|
||||
const hasRegressions = totalRegressions > 0;
|
||||
const verdictIcon = hasRegressions ? '\u274C' : '\u2705';
|
||||
let verdictText;
|
||||
if (hasRegressions && totalImprovements > 0) {
|
||||
verdictText = `${totalRegressions} regression(s), ${totalImprovements} improvement(s)`;
|
||||
} else if (hasRegressions) {
|
||||
verdictText = `${totalRegressions} regression(s) detected`;
|
||||
} else if (totalImprovements > 0) {
|
||||
verdictText = `No regressions \u2014 ${totalImprovements} improvement(s)`;
|
||||
} else {
|
||||
verdictText = 'No significant changes';
|
||||
}
|
||||
|
||||
lines.push(`# ${verdictIcon} Chat Performance: ${verdictText}`);
|
||||
lines.push('');
|
||||
lines.push(`| | |`);
|
||||
lines.push(`|---|---|`);
|
||||
lines.push(`| **Baseline** | ${baseLink} |`);
|
||||
lines.push(`| **Test** | ${testLink} |`);
|
||||
if (compareLink) {
|
||||
lines.push(`| **Diff** | ${compareLink} |`);
|
||||
}
|
||||
lines.push(`| **Runs per scenario** | ${opts.runs} |`);
|
||||
const overrides = Object.entries(opts.metricThresholds || {}).filter(([, v]) => {
|
||||
const parsed = typeof v === 'number' ? { type: 'fraction', value: v } : { type: 'absolute', value: parseFloat(/** @type {string} */(v)) };
|
||||
return parsed.type !== 'fraction' || parsed.value !== opts.threshold;
|
||||
});
|
||||
if (overrides.length > 0) {
|
||||
const overrideStr = overrides.map(([k, v]) => {
|
||||
if (typeof v === 'number') {
|
||||
return `${k}: ${(v * 100).toFixed(0)}%`;
|
||||
}
|
||||
return `${k}: ${v}`;
|
||||
}).join(', ');
|
||||
lines.push(`| **Regression threshold** | ${(opts.threshold * 100).toFixed(0)}% (${overrideStr}) |`);
|
||||
} else {
|
||||
lines.push(`| **Regression threshold** | ${(opts.threshold * 100).toFixed(0)}% |`);
|
||||
}
|
||||
lines.push(`| **Scenarios** | ${scenarios.length} |`);
|
||||
lines.push(`| **Platform** | ${jsonReport.platform || 'linux'} / x64 |`);
|
||||
lines.push('');
|
||||
|
||||
// -- Overview table --------------------------------------------------
|
||||
lines.push('## Overview');
|
||||
lines.push('');
|
||||
lines.push('| Scenario | TTFT | Complete | Layouts | Styles | LoAF | Verdict |');
|
||||
lines.push('|----------|-----:|---------:|--------:|-------:|-----:|:-------:|');
|
||||
|
||||
for (const scenario of scenarios) {
|
||||
const verdicts = scenarioVerdicts.get(scenario) || [];
|
||||
const get = (/** @type {string} */ m) => verdicts.find(v => v.metric === m);
|
||||
|
||||
const ttft = get('timeToFirstToken');
|
||||
const complete = get('timeToComplete');
|
||||
const layouts = get('layoutCount');
|
||||
const styles = get('recalcStyleCount');
|
||||
const loaf = get('longAnimationFrameCount');
|
||||
|
||||
const fmtCell = (/** @type {{ change: number } | undefined} */ v) => {
|
||||
if (!v) { return '\u2014'; }
|
||||
return `${v.change > 0 ? '+' : ''}${(v.change * 100).toFixed(0)}%`;
|
||||
};
|
||||
|
||||
const keyVerdicts = [ttft, complete, layouts, styles, loaf].filter(Boolean);
|
||||
const hasRegression = keyVerdicts.some(v => v?.verdict === 'REGRESSION');
|
||||
const hasImproved = keyVerdicts.some(v => v?.verdict === 'improved');
|
||||
const rowVerdict = hasRegression ? '\u274C' : hasImproved ? '\u2B06\uFE0F' : '\u2705';
|
||||
|
||||
lines.push(`| ${scenario} | ${fmtCell(ttft)} | ${fmtCell(complete)} | ${fmtCell(layouts)} | ${fmtCell(styles)} | ${fmtCell(loaf)} | ${rowVerdict} |`);
|
||||
}
|
||||
lines.push('');
|
||||
|
||||
// -- Regressions & improvements (compact table) ----------------------
|
||||
const notableRows = [];
|
||||
for (const scenario of scenarios) {
|
||||
const verdicts = scenarioVerdicts.get(scenario) || [];
|
||||
for (const v of verdicts) {
|
||||
if (v.verdict === 'REGRESSION' || v.verdict === 'improved') {
|
||||
notableRows.push({ scenario, ...v });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (notableRows.length > 0) {
|
||||
lines.push('## Regressions & Improvements');
|
||||
lines.push('');
|
||||
|
||||
lines.push('| Scenario | Metric | Baseline | Test | Change | p-value | |');
|
||||
lines.push('|----------|--------|----------|------|-------:|--------:|:-:|');
|
||||
for (const r of notableRows) {
|
||||
const pct = `${r.change > 0 ? '+' : ''}${(r.change * 100).toFixed(1)}%`;
|
||||
const icon = r.verdict === 'REGRESSION' ? '\u274C' : '\u2B06\uFE0F';
|
||||
lines.push(`| ${r.scenario} | ${r.metric} | ${r.basStr} | ${r.curStr} | ${pct} | ${r.pValue} | ${icon} |`);
|
||||
}
|
||||
lines.push('');
|
||||
}
|
||||
|
||||
// -- Full details (collapsible) --------------------------------------
|
||||
lines.push('<details><summary>Full metric details per scenario</summary>');
|
||||
lines.push('');
|
||||
|
||||
for (const scenario of scenarios) {
|
||||
const verdicts = scenarioVerdicts.get(scenario) || [];
|
||||
const base = baseline?.scenarios?.[scenario];
|
||||
|
||||
lines.push(`### ${scenario}`);
|
||||
lines.push('');
|
||||
|
||||
if (!base) {
|
||||
const current = jsonReport.scenarios[scenario];
|
||||
lines.push('> No baseline data for this scenario.');
|
||||
lines.push('');
|
||||
lines.push('| Metric | Value | StdDev | CV | n |');
|
||||
lines.push('|--------|------:|-------:|---:|--:|');
|
||||
for (const [metric, group, unit] of allMetrics) {
|
||||
const cur = current[group]?.[metric];
|
||||
if (!cur) { continue; }
|
||||
lines.push(`| ${metric} | ${cur.median}${unit} | \xb1${cur.stddev}${unit} | ${(cur.cv * 100).toFixed(0)}% | ${cur.n} |`);
|
||||
}
|
||||
lines.push('');
|
||||
continue;
|
||||
}
|
||||
|
||||
lines.push('| Metric | Baseline | Test | Change | p-value | Verdict |');
|
||||
lines.push('|--------|----------|------|--------|---------|---------|');
|
||||
for (const v of verdicts) {
|
||||
const pct = `${v.change > 0 ? '+' : ''}${(v.change * 100).toFixed(1)}%`;
|
||||
let verdictDisplay = v.verdict;
|
||||
if (v.verdict === 'REGRESSION') { verdictDisplay = '\u274C REGRESSION'; }
|
||||
else if (v.verdict === 'improved') { verdictDisplay = '\u2B06\uFE0F improved'; }
|
||||
else if (v.verdict === 'ok') { verdictDisplay = '\u2705 ok'; }
|
||||
else if (v.verdict === 'noise') { verdictDisplay = '\uD83C\uDF2B\uFE0F noise'; }
|
||||
else if (v.verdict === 'info') { verdictDisplay = '\u2139\uFE0F'; }
|
||||
lines.push(`| ${v.metric} | ${v.basStr} | ${v.curStr} | ${pct} | ${v.pValue} | ${verdictDisplay} |`);
|
||||
}
|
||||
lines.push('');
|
||||
}
|
||||
lines.push('</details>');
|
||||
lines.push('');
|
||||
|
||||
// -- Raw run data (collapsible) --------------------------------------
|
||||
lines.push('<details><summary>Raw run data</summary>');
|
||||
lines.push('');
|
||||
for (const scenario of scenarios) {
|
||||
const current = jsonReport.scenarios[scenario];
|
||||
lines.push(`### ${scenario}`);
|
||||
lines.push('');
|
||||
lines.push('| Run | TTFT (ms) | Complete (ms) | Layouts | Style Recalcs | LoAF Count | LoAF (ms) | Frames | Heap Delta (MB) |');
|
||||
lines.push('|----:|----------:|--------------:|--------:|--------------:|-----------:|----------:|-------:|----------------:|');
|
||||
const runs = current.rawRuns || [];
|
||||
for (let i = 0; i < runs.length; i++) {
|
||||
const r = runs[i];
|
||||
lines.push(`| ${i + 1} | ${round2(r.timeToFirstToken)} | ${r.timeToComplete} | ${r.layoutCount} | ${r.recalcStyleCount} | ${r.longAnimationFrameCount ?? '-'} | ${round2(r.longAnimationFrameTotalMs ?? 0) || '-'} | ${r.frameCount ?? '-'} | ${r.heapDelta} |`);
|
||||
}
|
||||
lines.push('');
|
||||
}
|
||||
if (baseline) {
|
||||
for (const scenario of scenarios) {
|
||||
const base = baseline.scenarios?.[scenario];
|
||||
if (!base) { continue; }
|
||||
lines.push(`### ${scenario} (baseline)`);
|
||||
lines.push('');
|
||||
lines.push('| Run | TTFT (ms) | Complete (ms) | Layouts | Style Recalcs | LoAF Count | LoAF (ms) | Frames | Heap Delta (MB) |');
|
||||
lines.push('|----:|----------:|--------------:|--------:|--------------:|-----------:|----------:|-------:|----------------:|');
|
||||
const runs = base.rawRuns || [];
|
||||
for (let i = 0; i < runs.length; i++) {
|
||||
const r = runs[i];
|
||||
lines.push(`| ${i + 1} | ${round2(r.timeToFirstToken)} | ${r.timeToComplete} | ${r.layoutCount} | ${r.recalcStyleCount} | ${r.longAnimationFrameCount ?? '-'} | ${round2(r.longAnimationFrameTotalMs ?? 0) || '-'} | ${r.frameCount ?? '-'} | ${r.heapDelta} |`);
|
||||
}
|
||||
lines.push('');
|
||||
}
|
||||
}
|
||||
lines.push('</details>');
|
||||
lines.push('');
|
||||
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
// -- Main --------------------------------------------------------------------
|
||||
|
||||
function main() {
|
||||
const opts = parseArgs();
|
||||
const merged = mergeResults(opts.resultsDir);
|
||||
|
||||
if (!merged) {
|
||||
const fallback = '\u26A0\uFE0F No perf results found to merge. Check perf-output.log artifacts.\n';
|
||||
fs.writeFileSync(opts.output, fallback);
|
||||
console.log('[merge] No results found.');
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const { report, baseline, baselineBuildVersion } = merged;
|
||||
const scenarioCount = Object.keys(report.scenarios).length;
|
||||
console.log(`[merge] Merged ${scenarioCount} scenarios from ${fs.readdirSync(opts.resultsDir).filter(d => d.startsWith('perf-results-')).length} groups`);
|
||||
if (baseline) {
|
||||
console.log(`[merge] Baseline: ${baselineBuildVersion || 'unknown'} (${Object.keys(baseline.scenarios).length} scenarios)`);
|
||||
}
|
||||
|
||||
const summary = generateUnifiedSummary(report, baseline, {
|
||||
threshold: merged.threshold || opts.threshold,
|
||||
metricThresholds: merged.metricThresholds,
|
||||
runs: report.runsPerScenario,
|
||||
baselineBuild: baselineBuildVersion,
|
||||
build: process.env.TEST_COMMIT || undefined,
|
||||
});
|
||||
|
||||
// Append leak summary if available
|
||||
let fullSummary = summary;
|
||||
if (opts.leakSummary && fs.existsSync(opts.leakSummary)) {
|
||||
fullSummary += '\n' + fs.readFileSync(opts.leakSummary, 'utf-8');
|
||||
console.log('[merge] Appended leak summary');
|
||||
}
|
||||
|
||||
fs.writeFileSync(opts.output, fullSummary);
|
||||
console.log(`[merge] Summary written to ${opts.output}`);
|
||||
}
|
||||
|
||||
main();
|
||||
@@ -0,0 +1,466 @@
|
||||
/*---------------------------------------------------------------------------------------------
|
||||
* Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
* Licensed under the MIT License. See License.txt in the project root for license information.
|
||||
*--------------------------------------------------------------------------------------------*/
|
||||
|
||||
// @ts-check
|
||||
|
||||
/**
|
||||
* Chat memory leak checker — state-based approach.
|
||||
*
|
||||
* The idea: if you return to the same state you started from, memory should
|
||||
* return to roughly the same level. Any residual growth is a potential leak.
|
||||
*
|
||||
* Each iteration:
|
||||
* 1. Open a fresh chat (baseline state)
|
||||
* 2. Measure heap + DOM nodes
|
||||
* 3. Cycle through ALL registered perf scenarios (text, code blocks,
|
||||
* tool calls, thinking, multi-turn, etc.)
|
||||
* 4. Open a new chat (return to baseline state — clears previous session)
|
||||
* 5. Measure heap + DOM nodes again
|
||||
* 6. The delta is the "leaked" memory for that iteration
|
||||
*
|
||||
* Multiple iterations let us detect consistent leaks vs. one-time caching.
|
||||
*
|
||||
* Usage:
|
||||
* npm run perf:chat-leak # defaults from config
|
||||
* npm run perf:chat-leak -- --iterations 5 # more iterations
|
||||
* npm run perf:chat-leak -- --threshold 5 # 5MB total threshold
|
||||
* npm run perf:chat-leak -- --build 1.115.0 # test a specific build
|
||||
*/
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const {
|
||||
DATA_DIR, loadConfig,
|
||||
resolveBuild, buildEnv, buildArgs, prepareRunDir,
|
||||
launchVSCode,
|
||||
} = require('./common/utils');
|
||||
const {
|
||||
CONTENT_SCENARIOS, TOOL_CALL_SCENARIOS, MULTI_TURN_SCENARIOS,
|
||||
} = require('./common/perf-scenarios');
|
||||
const {
|
||||
getUserTurns, getModelTurnCount,
|
||||
} = require('./common/mock-llm-server');
|
||||
|
||||
// -- Config (edit config.jsonc to change defaults) ---------------------------
|
||||
|
||||
const CONFIG = loadConfig('memLeaks');
|
||||
|
||||
// -- CLI args ----------------------------------------------------------------
|
||||
|
||||
function parseArgs() {
|
||||
const args = process.argv.slice(2);
|
||||
const opts = {
|
||||
iterations: CONFIG.iterations ?? 3,
|
||||
messages: CONFIG.messages ?? 5,
|
||||
verbose: false,
|
||||
ci: false,
|
||||
/** @type {string | undefined} */
|
||||
build: undefined,
|
||||
leakThresholdMB: CONFIG.leakThresholdMB ?? 5,
|
||||
/** @type {Record<string, any>} */
|
||||
settingsOverrides: {},
|
||||
};
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
switch (args[i]) {
|
||||
case '--iterations': opts.iterations = parseInt(args[++i], 10); break;
|
||||
case '--messages': case '-n': opts.messages = parseInt(args[++i], 10); break;
|
||||
case '--verbose': opts.verbose = true; break;
|
||||
case '--ci': opts.ci = true; break;
|
||||
case '--build': case '-b': opts.build = args[++i]; break;
|
||||
case '--threshold': opts.leakThresholdMB = parseFloat(args[++i]); break;
|
||||
case '--setting': {
|
||||
const kv = args[++i];
|
||||
const eq = kv.indexOf('=');
|
||||
if (eq === -1) { console.error(`--setting requires key=value, got: ${kv}`); process.exit(1); }
|
||||
const key = kv.slice(0, eq);
|
||||
const raw = kv.slice(eq + 1);
|
||||
const val = raw === 'true' ? true : raw === 'false' ? false : /^-?\d+(\.\d+)?$/.test(raw) ? Number(raw) : raw;
|
||||
opts.settingsOverrides[key] = val;
|
||||
break;
|
||||
}
|
||||
case '--help': case '-h':
|
||||
console.log([
|
||||
'Chat memory leak checker (state-based)',
|
||||
'',
|
||||
'Options:',
|
||||
' --iterations <n> Number of open→work→reset cycles (default: 3)',
|
||||
' --messages <n> Messages to send per iteration (default: 5)',
|
||||
' --ci CI mode: write Markdown summary to ci-summary.md',
|
||||
' --build <path|ver> Path to VS Code build or version to download',
|
||||
' --threshold <MB> Max total residual heap growth in MB (default: 5)',
|
||||
' --setting <k=v> Set a VS Code setting override (repeatable)',
|
||||
' --verbose Print per-step details',
|
||||
].join('\n'));
|
||||
process.exit(0);
|
||||
}
|
||||
}
|
||||
return opts;
|
||||
}
|
||||
|
||||
// -- Scenario list -----------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Build a flat list of scenario IDs to cycle through during leak testing.
|
||||
* Includes all scenario types: content-only, tool-call, and multi-turn.
|
||||
*
|
||||
* Content scenarios exercise varied rendering (code blocks, markdown, etc.).
|
||||
* Tool-call scenarios exercise the agent loop (model → tool → model → ...).
|
||||
* Multi-turn scenarios exercise user follow-ups and thinking blocks.
|
||||
*/
|
||||
function getScenarioIds() {
|
||||
return [
|
||||
...Object.keys(CONTENT_SCENARIOS),
|
||||
...Object.keys(TOOL_CALL_SCENARIOS),
|
||||
...Object.keys(MULTI_TURN_SCENARIOS),
|
||||
];
|
||||
}
|
||||
|
||||
// -- Helpers -----------------------------------------------------------------
|
||||
|
||||
const CHAT_VIEW = 'div[id="workbench.panel.chat"]';
|
||||
const CHAT_EDITOR_SEL = `${CHAT_VIEW} .interactive-input-part .monaco-editor[role="code"]`;
|
||||
|
||||
/**
|
||||
* Measure heap (MB) and DOM node count after forced GC.
|
||||
* @param {any} cdp
|
||||
* @param {import('playwright').Page} page
|
||||
*/
|
||||
async function measure(cdp, page) {
|
||||
await cdp.send('HeapProfiler.collectGarbage');
|
||||
await new Promise(r => setTimeout(r, 500));
|
||||
await cdp.send('HeapProfiler.collectGarbage');
|
||||
await new Promise(r => setTimeout(r, 300));
|
||||
const heapInfo = /** @type {any} */ (await cdp.send('Runtime.getHeapUsage'));
|
||||
const heapMB = Math.round(heapInfo.usedSize / 1024 / 1024 * 100) / 100;
|
||||
const domNodes = await page.evaluate(() => document.querySelectorAll('*').length);
|
||||
return { heapMB, domNodes };
|
||||
}
|
||||
|
||||
/**
|
||||
* Open a new chat session via the command palette.
|
||||
* @param {import('playwright').Page} page
|
||||
*/
|
||||
async function openNewChat(page) {
|
||||
// Use keyboard shortcut to open a new chat (clears previous session)
|
||||
const newChatShortcut = process.platform === 'darwin' ? 'Meta+KeyL' : 'Control+KeyL';
|
||||
await page.keyboard.press(newChatShortcut);
|
||||
await new Promise(r => setTimeout(r, 1000));
|
||||
|
||||
// Verify the chat view is visible and ready
|
||||
await page.waitForSelector(CHAT_VIEW, { timeout: 15_000 });
|
||||
await page.waitForFunction(
|
||||
(sel) => Array.from(document.querySelectorAll(sel)).some(el => el.getBoundingClientRect().width > 0),
|
||||
CHAT_EDITOR_SEL, { timeout: 15_000 },
|
||||
);
|
||||
await new Promise(r => setTimeout(r, 500));
|
||||
}
|
||||
|
||||
/**
|
||||
* Send a single message and wait for the response to complete.
|
||||
* For multi-turn scenarios where the model makes multiple tool-call rounds
|
||||
* before producing content, `modelTurns` controls how many completions to
|
||||
* wait for.
|
||||
* @param {import('playwright').Page} page
|
||||
* @param {{ completionCount: () => number, waitForCompletion: (n: number, ms: number) => Promise<void> }} mockServer
|
||||
* @param {string} text
|
||||
* @param {number} [modelTurns=1] - number of model completions to wait for
|
||||
*/
|
||||
async function sendMessage(page, mockServer, text, modelTurns = 1) {
|
||||
await page.click(CHAT_EDITOR_SEL);
|
||||
await new Promise(r => setTimeout(r, 200));
|
||||
|
||||
const inputSel = await page.evaluate((editorSel) => {
|
||||
const ed = document.querySelector(editorSel);
|
||||
if (!ed) { throw new Error('no editor'); }
|
||||
return ed.querySelector('.native-edit-context') ? editorSel + ' .native-edit-context' : editorSel + ' textarea';
|
||||
}, CHAT_EDITOR_SEL);
|
||||
|
||||
const hasDriver = await page.evaluate(() =>
|
||||
// @ts-ignore
|
||||
!!globalThis.driver?.typeInEditor
|
||||
).catch(() => false);
|
||||
|
||||
if (hasDriver) {
|
||||
await page.evaluate(({ selector, t }) => {
|
||||
// @ts-ignore
|
||||
return globalThis.driver.typeInEditor(selector, t);
|
||||
}, { selector: inputSel, t: text });
|
||||
} else {
|
||||
await page.click(inputSel);
|
||||
await new Promise(r => setTimeout(r, 200));
|
||||
await page.locator(inputSel).pressSequentially(text, { delay: 0 });
|
||||
}
|
||||
|
||||
const compBefore = mockServer.completionCount();
|
||||
await page.keyboard.press('Enter');
|
||||
try { await mockServer.waitForCompletion(compBefore + modelTurns, 60_000); } catch { }
|
||||
|
||||
const responseSelector = `${CHAT_VIEW} .interactive-item-container.interactive-response`;
|
||||
await page.waitForFunction(
|
||||
(sel) => {
|
||||
const responses = document.querySelectorAll(sel);
|
||||
if (responses.length === 0) { return false; }
|
||||
return !responses[responses.length - 1].classList.contains('chat-response-loading');
|
||||
},
|
||||
responseSelector, { timeout: 30_000 },
|
||||
);
|
||||
await new Promise(r => setTimeout(r, 500));
|
||||
}
|
||||
|
||||
/**
|
||||
* Run a full scenario: send the initial message, then handle any user
|
||||
* follow-up turns for multi-turn scenarios.
|
||||
*
|
||||
* - Content-only scenarios: single message, 1 model turn.
|
||||
* - Tool-call scenarios (no user turns): single message, N model turns
|
||||
* (the extension automatically relays tool results back to the model).
|
||||
* - Multi-turn with user turns: send initial message, wait for response,
|
||||
* then for each user turn send the follow-up message and wait again.
|
||||
*
|
||||
* @param {import('playwright').Page} page
|
||||
* @param {{ completionCount: () => number, waitForCompletion: (n: number, ms: number) => Promise<void> }} mockServer
|
||||
* @param {string} scenarioId
|
||||
* @param {string} label - prefix for the message (e.g. "Warmup" or "Iteration 2")
|
||||
*/
|
||||
async function runScenario(page, mockServer, scenarioId, label) {
|
||||
const userTurns = getUserTurns(scenarioId);
|
||||
const totalModelTurns = getModelTurnCount(scenarioId);
|
||||
|
||||
if (userTurns.length === 0) {
|
||||
// Content-only or tool-call scenario: one message, wait for all model turns
|
||||
await sendMessage(page, mockServer, `[scenario:${scenarioId}] ${label}`, totalModelTurns);
|
||||
} else {
|
||||
// Multi-turn with user follow-ups: send initial message and wait for
|
||||
// the model turns before the first user turn, then alternate.
|
||||
let modelTurnsSoFar = 0;
|
||||
const firstUserAfter = userTurns[0].afterModelTurn;
|
||||
const turnsBeforeFirstUser = firstUserAfter - modelTurnsSoFar;
|
||||
await sendMessage(page, mockServer, `[scenario:${scenarioId}] ${label}`, turnsBeforeFirstUser);
|
||||
modelTurnsSoFar = firstUserAfter;
|
||||
|
||||
for (let u = 0; u < userTurns.length; u++) {
|
||||
const nextModelStop = u + 1 < userTurns.length
|
||||
? userTurns[u + 1].afterModelTurn
|
||||
: totalModelTurns;
|
||||
const turnsUntilNext = nextModelStop - modelTurnsSoFar;
|
||||
|
||||
// Send the user follow-up message
|
||||
await sendMessage(page, mockServer, userTurns[u].message, turnsUntilNext);
|
||||
modelTurnsSoFar = nextModelStop;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// -- Leak check --------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* @param {string} electronPath
|
||||
* @param {{ url: string, requestCount: () => number, waitForRequests: (n: number, ms: number) => Promise<void>, completionCount: () => number, waitForCompletion: (n: number, ms: number) => Promise<void> }} mockServer
|
||||
* @param {{ iterations: number, verbose: boolean, settingsOverrides?: Record<string, any> }} opts
|
||||
*/
|
||||
async function runLeakCheck(electronPath, mockServer, opts) {
|
||||
const { iterations, verbose } = opts;
|
||||
const { userDataDir, extDir, logsDir } = prepareRunDir('leak-check', mockServer, opts.settingsOverrides);
|
||||
const isDevBuild = !electronPath.includes('.vscode-test');
|
||||
|
||||
const vscode = await launchVSCode(
|
||||
electronPath,
|
||||
buildArgs(userDataDir, extDir, logsDir, { isDevBuild }),
|
||||
buildEnv(mockServer, { isDevBuild }),
|
||||
{ verbose },
|
||||
);
|
||||
const page = vscode.page;
|
||||
|
||||
try {
|
||||
await page.waitForSelector('.monaco-workbench', { timeout: 60_000 });
|
||||
|
||||
const cdp = await page.context().newCDPSession(page);
|
||||
await cdp.send('HeapProfiler.enable');
|
||||
|
||||
// Open chat panel
|
||||
const chatShortcut = process.platform === 'darwin' ? 'Control+Meta+KeyI' : 'Control+Alt+KeyI';
|
||||
await page.keyboard.press(chatShortcut);
|
||||
await page.waitForSelector(CHAT_VIEW, { timeout: 15_000 });
|
||||
await page.waitForFunction(
|
||||
(sel) => Array.from(document.querySelectorAll(sel)).some(el => el.getBoundingClientRect().width > 0),
|
||||
CHAT_EDITOR_SEL, { timeout: 15_000 },
|
||||
);
|
||||
|
||||
// Wait for extension activation
|
||||
const reqsBefore = mockServer.requestCount();
|
||||
try { await mockServer.waitForRequests(reqsBefore + 4, 30_000); } catch { }
|
||||
await new Promise(r => setTimeout(r, 3000));
|
||||
|
||||
const scenarioIds = getScenarioIds();
|
||||
|
||||
// --- Baseline measurement (fresh chat) ---
|
||||
const baseline = await measure(cdp, page);
|
||||
if (verbose) {
|
||||
console.log(` [leak] Baseline: heap=${baseline.heapMB}MB, domNodes=${baseline.domNodes}`);
|
||||
}
|
||||
|
||||
/** @type {{ beforeHeapMB: number, afterHeapMB: number, deltaHeapMB: number, beforeDomNodes: number, afterDomNodes: number, deltaDomNodes: number }[]} */
|
||||
const iterationResults = [];
|
||||
|
||||
for (let iter = 0; iter < iterations; iter++) {
|
||||
// Measure at start of iteration (should be in "clean" state)
|
||||
const before = await measure(cdp, page);
|
||||
|
||||
if (verbose) {
|
||||
console.log(` [leak] Iteration ${iter + 1}/${iterations}: start heap=${before.heapMB}MB, domNodes=${before.domNodes}`);
|
||||
}
|
||||
|
||||
// Do work: cycle through all scenarios
|
||||
for (let m = 0; m < scenarioIds.length; m++) {
|
||||
const sid = scenarioIds[m];
|
||||
await runScenario(page, mockServer, sid, `Iteration ${iter + 1}`);
|
||||
if (verbose) {
|
||||
console.log(` [leak] Sent ${sid} (${m + 1}/${scenarioIds.length})`);
|
||||
}
|
||||
}
|
||||
|
||||
// Return to clean state: open a new empty chat
|
||||
await openNewChat(page);
|
||||
await new Promise(r => setTimeout(r, 1000));
|
||||
|
||||
// Measure after returning to clean state
|
||||
const after = await measure(cdp, page);
|
||||
const deltaHeapMB = Math.round((after.heapMB - before.heapMB) * 100) / 100;
|
||||
const deltaDomNodes = after.domNodes - before.domNodes;
|
||||
|
||||
iterationResults.push({
|
||||
beforeHeapMB: before.heapMB,
|
||||
afterHeapMB: after.heapMB,
|
||||
deltaHeapMB,
|
||||
beforeDomNodes: before.domNodes,
|
||||
afterDomNodes: after.domNodes,
|
||||
deltaDomNodes,
|
||||
});
|
||||
|
||||
if (verbose) {
|
||||
console.log(` [leak] Iteration ${iter + 1}/${iterations}: end heap=${after.heapMB}MB (delta=${deltaHeapMB}MB), domNodes=${after.domNodes} (delta=${deltaDomNodes})`);
|
||||
}
|
||||
}
|
||||
|
||||
// Final measurement
|
||||
const final = await measure(cdp, page);
|
||||
const totalResidualMB = Math.round((final.heapMB - baseline.heapMB) * 100) / 100;
|
||||
const totalResidualNodes = final.domNodes - baseline.domNodes;
|
||||
|
||||
return {
|
||||
baseline,
|
||||
final: { heapMB: final.heapMB, domNodes: final.domNodes },
|
||||
totalResidualMB,
|
||||
totalResidualNodes,
|
||||
iterations: iterationResults,
|
||||
};
|
||||
} finally {
|
||||
await vscode.close();
|
||||
}
|
||||
}
|
||||
|
||||
// -- Main --------------------------------------------------------------------
|
||||
|
||||
async function main() {
|
||||
const opts = parseArgs();
|
||||
const electronPath = await resolveBuild(opts.build);
|
||||
|
||||
if (!fs.existsSync(electronPath)) {
|
||||
console.error(`Electron not found at: ${electronPath}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const { startServer } = require('./common/mock-llm-server');
|
||||
const { registerPerfScenarios } = require('./common/perf-scenarios');
|
||||
registerPerfScenarios();
|
||||
const mockServer = await startServer(0);
|
||||
|
||||
console.log(`[chat-simulation] Leak check: ${opts.iterations} iterations × ${getScenarioIds().length} scenarios, threshold ${opts.leakThresholdMB}MB total`);
|
||||
console.log(`[chat-simulation] Build: ${electronPath}`);
|
||||
console.log('');
|
||||
|
||||
const result = await runLeakCheck(electronPath, mockServer, opts);
|
||||
|
||||
console.log('[chat-simulation] =================== Leak Check Results ===================');
|
||||
console.log('');
|
||||
console.log(` Baseline: heap=${result.baseline.heapMB}MB, domNodes=${result.baseline.domNodes}`);
|
||||
console.log(` Final: heap=${result.final.heapMB}MB, domNodes=${result.final.domNodes}`);
|
||||
console.log('');
|
||||
for (let i = 0; i < result.iterations.length; i++) {
|
||||
const it = result.iterations[i];
|
||||
console.log(` Iteration ${i + 1}: ${it.beforeHeapMB}MB → ${it.afterHeapMB}MB (residual: ${it.deltaHeapMB > 0 ? '+' : ''}${it.deltaHeapMB}MB, DOM: ${it.deltaDomNodes > 0 ? '+' : ''}${it.deltaDomNodes} nodes)`);
|
||||
}
|
||||
console.log('');
|
||||
console.log(` Total residual heap growth: ${result.totalResidualMB > 0 ? '+' : ''}${result.totalResidualMB}MB`);
|
||||
console.log(` Total residual DOM growth: ${result.totalResidualNodes > 0 ? '+' : ''}${result.totalResidualNodes} nodes`);
|
||||
console.log('');
|
||||
|
||||
// Write JSON
|
||||
const jsonPath = path.join(DATA_DIR, 'chat-simulation-leak-results.json');
|
||||
fs.writeFileSync(jsonPath, JSON.stringify({
|
||||
timestamp: new Date().toISOString(),
|
||||
leakThresholdMB: opts.leakThresholdMB,
|
||||
iterationCount: opts.iterations,
|
||||
scenarioCount: getScenarioIds().length,
|
||||
...result,
|
||||
}, null, 2));
|
||||
console.log(`[chat-simulation] Results written to ${jsonPath}`);
|
||||
|
||||
const leaked = result.totalResidualMB > opts.leakThresholdMB;
|
||||
console.log('');
|
||||
if (leaked) {
|
||||
console.log(`[chat-simulation] LEAK DETECTED — ${result.totalResidualMB}MB residual exceeds ${opts.leakThresholdMB}MB threshold`);
|
||||
} else {
|
||||
console.log(`[chat-simulation] No leak detected (${result.totalResidualMB}MB residual < ${opts.leakThresholdMB}MB threshold)`);
|
||||
}
|
||||
|
||||
if (opts.ci) {
|
||||
const summary = generateLeakCISummary(result, opts);
|
||||
const summaryPath = path.join(DATA_DIR, 'ci-summary-leak.md');
|
||||
fs.writeFileSync(summaryPath, summary);
|
||||
console.log(`[chat-simulation] CI summary written to ${summaryPath}`);
|
||||
}
|
||||
|
||||
await mockServer.close();
|
||||
process.exit(leaked ? 1 : 0);
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate a Markdown summary for CI, matching the perf script pattern.
|
||||
* @param {{ baseline: { heapMB: number, domNodes: number }, final: { heapMB: number, domNodes: number }, totalResidualMB: number, totalResidualNodes: number, iterations: { beforeHeapMB: number, afterHeapMB: number, deltaHeapMB: number, beforeDomNodes: number, afterDomNodes: number, deltaDomNodes: number }[] }} result
|
||||
* @param {{ leakThresholdMB: number, iterations: number }} opts
|
||||
*/
|
||||
function generateLeakCISummary(result, opts) {
|
||||
const leaked = result.totalResidualMB > opts.leakThresholdMB;
|
||||
const verdict = leaked ? '\u274C **LEAK DETECTED**' : '\u2705 **No leak detected**';
|
||||
const lines = [];
|
||||
lines.push('## Memory Leak Check');
|
||||
lines.push('');
|
||||
lines.push('| | |');
|
||||
lines.push('|---|---|');
|
||||
lines.push(`| **Verdict** | ${verdict} |`);
|
||||
lines.push(`| **Threshold** | ${opts.leakThresholdMB} MB |`);
|
||||
lines.push(`| **Iterations** | ${opts.iterations} |`);
|
||||
lines.push(`| **Scenarios per iteration** | ${getScenarioIds().length} |`);
|
||||
lines.push('');
|
||||
lines.push('| Phase | Heap (MB) | DOM Nodes |');
|
||||
lines.push('|-------|----------:|----------:|');
|
||||
lines.push(`| Baseline | ${result.baseline.heapMB} | ${result.baseline.domNodes} |`);
|
||||
for (let i = 0; i < result.iterations.length; i++) {
|
||||
const it = result.iterations[i];
|
||||
const sign = it.deltaHeapMB > 0 ? '+' : '';
|
||||
const domSign = it.deltaDomNodes > 0 ? '+' : '';
|
||||
lines.push(`| Iteration ${i + 1} | ${it.afterHeapMB} (${sign}${it.deltaHeapMB}) | ${it.afterDomNodes} (${domSign}${it.deltaDomNodes}) |`);
|
||||
}
|
||||
lines.push(`| **Final** | **${result.final.heapMB}** | **${result.final.domNodes}** |`);
|
||||
lines.push('');
|
||||
const sign = result.totalResidualMB > 0 ? '+' : '';
|
||||
const domSign = result.totalResidualNodes > 0 ? '+' : '';
|
||||
lines.push(`**Total residual growth:** ${sign}${result.totalResidualMB} MB heap, ${domSign}${result.totalResidualNodes} DOM nodes`);
|
||||
lines.push('');
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
main().catch(err => { console.error(err); process.exit(1); });
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user