diff --git a/.github/skills/chat-perf/SKILL.md b/.github/skills/chat-perf/SKILL.md index 7769d511cf4..41c5e22524a 100644 --- a/.github/skills/chat-perf/SKILL.md +++ b/.github/skills/chat-perf/SKILL.md @@ -42,6 +42,8 @@ npm run perf:chat-leak -- --messages 20 --verbose Launches VS Code via Playwright Electron, opens the chat panel, sends a message with a mock LLM response, and measures timing, layout, and rendering metrics. By default, downloads VS Code 1.115.0 as a baseline, benchmarks it, then benchmarks the local dev build and compares. +> **You don't always need a baseline.** A baseline exists only for *comparison* (regression detection). If you just want the current build's numbers — profiling a single change, capturing traces/heap snapshots, or iterating on a scenario — pass `--no-baseline` to skip downloading and benchmarking the baseline entirely (roughly halves runtime). Baseline comparison is what turns raw measurements into a pass/fail verdict; without it you still get all the metrics, just no verdict. + ### Key flags | Flag | Default | Description | @@ -51,7 +53,7 @@ Launches VS Code via Playwright Electron, opens the chat panel, sends a message | `--build ` / `-b` | local dev | Build to test. Accepts path or version (`1.110.0`, `insiders`, commit hash). | | `--baseline ` | — | Compare against a previously saved baseline JSON file. | | `--baseline-build ` | `1.115.0` | Version or local path to benchmark as baseline. | -| `--no-baseline` | — | Skip baseline comparison entirely. | +| `--no-baseline` | — | Skip the baseline entirely — just measure the test build (no download, no comparison, ~2× faster). Use when you only need raw numbers, not a regression verdict. | | `--save-baseline` | — | Save results as the new baseline (requires `--baseline `). | | `--resume ` | — | Resume a previous run, adding more iterations to increase confidence. | | `--threshold ` | `0.2` | Regression threshold (0.2 = flag if 20% slower). | @@ -60,6 +62,7 @@ Launches VS Code via Playwright Electron, opens the chat panel, sends a message | `--force` | — | Skip build mode mismatch confirmation prompt. | | `--ci` | — | CI mode: write Markdown summary to `ci-summary.md` (implies `--no-cache`, `--heap-snapshots`, `--cleanup-diagnostics`). | | `--heap-snapshots` | — | Take heap snapshots after each run (slow; auto-enabled in `--ci` mode). | +| `--gc-object-stats` | — | **GC deep-dives only.** Enables V8 `gc_stats` tracing (per-type heap object dump on every GC). ⚠️ Corrupts all timing metrics — a major GC landing mid-request adds ~550ms — so never use it for benchmarking. Off by default; prefer heap snapshots for memory analysis. | | `--cleanup-diagnostics` | — | Delete heap snapshots, CPU profiles, and traces to save disk. During runs, only the latest run's files are kept; after comparison, files for non-regressed scenarios are deleted. Auto-enabled in `--ci` mode. | | `--setting ` | — | Set a VS Code setting override for all builds (repeatable). | | `--test-setting ` | — | Set a VS Code setting override for the test build only. | @@ -186,13 +189,13 @@ Run `npm run perf:chat -- --help` to see the full list of registered scenario ID Only these metrics trigger a regression failure (when they exceed the threshold with statistical significance): - `timeToFirstToken`, `timeToComplete` — user-perceived latency +- `layoutDurationMs` — total layout time from the trace (the *real* layout cost) - `forcedReflowCount` — forced synchronous layouts are always bad - `longTaskCount`, `longAnimationFrameCount` — main thread jank These are reported but **informational only** (won't fail CI): -- `layoutCount` — inflated by CSS animations; use `layoutDurationMs` instead -- `layoutDurationMs` — total layout time from trace (more meaningful than count) -- `recalcStyleCount` — inflated by CSS animations (compositor-driven, cheap) +- `layoutCount` — number of layout ops; inflated by CSS animations (compositor-driven, cheap). A build can do *more but cheaper* layouts, so gate on `layoutDurationMs`, not this count. +- `recalcStyleCount` — number of style recalcs; inflated by CSS animations (compositor-driven, cheap) - `timeToRenderComplete` — includes typewriter animation tail - Memory/heap metrics — too noisy for single-request benchmarks @@ -229,6 +232,54 @@ Launches one VS Code session, sends N messages sequentially, forces GC between e - DOM nodes stable after first message — normal (chat list virtualization working) - DOM nodes growing linearly — rendering leak, check disposable cleanup +## CI runs & pinpointing regressions + +The perf + leak checks run in CI via the **`.github/workflows/chat-perf.yml`** workflow (a scheduled daily `workflow_dispatch`, plus manual dispatch). Each run benchmarks the current `main` as the **test** build against a fixed release **baseline** (from `config.jsonc`, e.g. `1.122.0`). Because the baseline is fixed, the **test** median for a metric across successive daily runs traces `main`'s trajectory over time — that's what lets you bisect *when* something regressed or went flaky. + +### Finding and reading historic runs + +```bash +# List recent runs (most recent first) — note the run IDs and dates +gh run list --workflow chat-perf.yml -R microsoft/vscode --limit 30 \ + --json databaseId,status,conclusion,createdAt,headBranch \ + --jq '.[] | [.databaseId, (.conclusion//.status), .createdAt, .headBranch] | @tsv' + +# See a run's per-job results +gh run view -R microsoft/vscode --json jobs \ + --jq '.jobs[] | [.name, (.conclusion//.status)] | @tsv' + +# Trigger a run manually against any ref/version: +gh workflow run chat-perf.yml -R microsoft/vscode --ref main \ + -f test_build= -f baseline_build= +``` + +### Artifacts per run (and retention — this matters for old runs) + +| Artifact | Contents | Retention | +|---|---|---| +| `chat-perf-summary` | Unified `ci-summary.md`: verdicts, per-metric medians ±stddev, **per-run raw tables** | 30 days | +| `perf-results-` | Everything below **plus traces, CPU profiles, heap snapshots** (large) | 30 days | +| `leak-results` | Leak log + `chat-simulation-leak-results.json` | 30 days | +| `perf-summary-` | `results.json` (full per-run metrics incl. `rawRuns`) + `baseline-*.json` | **1 day** | + +```bash +# List a run's artifacts + whether they've expired +gh api repos/microsoft/vscode/actions/runs//artifacts \ + --jq '.artifacts[] | [.name, .expired] | @tsv' + +# Download the human-readable summary (best first stop; survives 30 days) +gh run download -R microsoft/vscode -n chat-perf-summary +``` + +### Pinpointing where a metric regressed / went flaky + +1. **Bisect across dates.** Pull `chat-perf-summary/ci-summary.md` from several runs spanning the window. Compare the **test** median (and ±stddev) for the suspect metric+scenario run-to-run — the day it jumps is when `main` changed. A metric that goes *bimodal* (e.g. a raw-run column showing two clusters like `~250 / ~900`) is the flaky signature; a stable shift is a genuine regression. +2. **Confirm it's real vs. measurement noise.** Check the per-run raw tables (in `ci-summary.md`) — high ±stddev / bimodal values mean the *median* is being pulled around by a few outlier runs, not a uniform slowdown. +3. **Deep-dive the cause.** Download `perf-results-` (has the `trace.json` per run) for a slow run and inspect what dominates the slow window. Trick that found the `gc_stats` artifact: sum main-thread `RunTask` durations between two `code/chat/*` marks (e.g. `willCollectInstructions` → `didCollectInstructions`) — if the window is ~0% busy, it's an async wait or a GC pause, not real work; then look at the largest `X`-phase events in that window (`MajorGC`, `Layout`, etc.). +4. **Reproduce a suspect commit locally** to bisect precisely: `npm run perf:chat -- --build --baseline-build --runs 7` (or two commits directly via `--build --baseline-build `). + +> Tip: `perf-summary-*` (the machine-readable `results.json` with `rawRuns`) is deleted after **1 day**, so for older runs rely on `chat-perf-summary` (raw tables, 30 days) or extract `results.json` from `perf-results-*` (also 30 days). + ## Architecture ``` diff --git a/.github/workflows/chat-perf.yml b/.github/workflows/chat-perf.yml index 8f67c576e02..8713e9a567e 100644 --- a/.github/workflows/chat-perf.yml +++ b/.github/workflows/chat-perf.yml @@ -57,13 +57,19 @@ env: SCENARIOS_INPUT: ${{ inputs.scenarios || '' }} TEST_SETTINGS_INPUT: ${{ inputs.test_settings || '' }} BASELINE_SETTINGS_INPUT: ${{ inputs.baseline_settings || '' }} + # Skip binary downloads during `npm ci` (they hang intermittently on hosted + # runners inside the nested postinstall installs). Electron and Playwright are + # fetched explicitly later via `Download Electron` and `Install Playwright + # Chromium`, which ignore these flags. Mirrors pr-node-modules.yml. + ELECTRON_SKIP_BINARY_DOWNLOAD: 1 + PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: 1 jobs: # ── Shared setup: build once, cache everything ────────────────────── setup: name: Build & Cache runs-on: ubuntu-latest - timeout-minutes: 30 + timeout-minutes: 50 outputs: test_is_version: ${{ steps.resolve.outputs.is_version }} test_build_arg: ${{ steps.resolve.outputs.build_arg }} @@ -111,13 +117,33 @@ jobs: libasound2t64 libxshmfence1 libgtk-3-0 - name: Install dependencies - run: npm ci + run: | + set -e + # Retry with a per-attempt timeout to survive transient npm registry + # hangs/failures on hosted runners (mirrors pr-node-modules.yml). + for i in {1..5}; do + timeout 900 npm ci && break + if [ $i -eq 5 ]; then + echo "npm ci failed/hung too many times" >&2 + exit 1 + fi + echo "npm ci attempt $i failed or timed out, retrying..." + done env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - name: Install build dependencies - run: npm ci working-directory: build + run: | + set -e + for i in {1..5}; do + timeout 900 npm ci && break + if [ $i -eq 5 ]; then + echo "npm ci failed/hung too many times" >&2 + exit 1 + fi + echo "npm ci attempt $i failed or timed out, retrying..." + done - name: Transpile source run: npm run transpile-client @@ -210,7 +236,16 @@ jobs: libasound2t64 libxshmfence1 libgtk-3-0 - name: Install dependencies - run: npm ci + run: | + set -e + for i in {1..5}; do + timeout 900 npm ci && break + if [ $i -eq 5 ]; then + echo "npm ci failed/hung too many times" >&2 + exit 1 + fi + echo "npm ci attempt $i failed or timed out, retrying..." + done env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} @@ -383,7 +418,16 @@ jobs: libasound2t64 libxshmfence1 libgtk-3-0 - name: Install dependencies - run: npm ci + run: | + set -e + for i in {1..5}; do + timeout 900 npm ci && break + if [ $i -eq 5 ]; then + echo "npm ci failed/hung too many times" >&2 + exit 1 + fi + echo "npm ci attempt $i failed or timed out, retrying..." + done env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/scripts/chat-simulation/common/utils.js b/scripts/chat-simulation/common/utils.js index 79c608cf2ec..e97f84f8c32 100644 --- a/scripts/chat-simulation/common/utils.js +++ b/scripts/chat-simulation/common/utils.js @@ -217,12 +217,23 @@ function buildEnv(mockServer, { isDevBuild = true } = {}) { * @param {string} logsDir * @returns {string[]} */ -function buildArgs(userDataDir, extDir, logsDir, { isDevBuild = true, extHostInspectPort = 0, traceFile = '', appRoot = ROOT } = {}) { +function buildArgs(userDataDir, extDir, logsDir, { isDevBuild = true, extHostInspectPort = 0, traceFile = '', appRoot = ROOT, gcObjectStats = false } = {}) { // Chromium switches must come BEFORE the app path (ROOT) — Chromium // only processes switches that precede the first non-switch argument. const chromiumFlags = []; if (traceFile) { - chromiumFlags.push(`--enable-tracing=v8.gc,disabled-by-default-v8.gc,disabled-by-default-v8.gc_stats,devtools.timeline,blink.user_timing`); + // IMPORTANT: `disabled-by-default-v8.gc_stats` is intentionally OFF by + // default. It makes V8 run GC_OBJECT_DUMP_STATISTICS (a full per-type + // heap object dump) on every major GC, inflating a ~15ms GC pause to + // ~550ms. When such a GC lands in the measured request window it + // corrupts timeToFirstToken (bimodal ~250ms vs ~900ms). `v8.gc` + + // `disabled-by-default-v8.gc` still provide the GC events we count. + // Opt in via `--gc-object-stats` only for deliberate GC deep-dives + // (never for timing runs), accepting that timings become unreliable. + const gcCategories = gcObjectStats + ? 'v8.gc,disabled-by-default-v8.gc,disabled-by-default-v8.gc_stats' + : 'v8.gc,disabled-by-default-v8.gc'; + chromiumFlags.push(`--enable-tracing=${gcCategories},devtools.timeline,blink.user_timing`); chromiumFlags.push(`--trace-startup-file=${traceFile}`); chromiumFlags.push(`--enable-tracing-format=json`); } diff --git a/scripts/chat-simulation/config.jsonc b/scripts/chat-simulation/config.jsonc index 6555b532e71..54a3dd96d27 100644 --- a/scripts/chat-simulation/config.jsonc +++ b/scripts/chat-simulation/config.jsonc @@ -17,8 +17,7 @@ "metricThresholds": { "timeToFirstToken": "100ms", "timeToComplete": 0.2, - "layoutCount": 0.2, - "recalcStyleCount": 0.2, + "layoutDurationMs": 0.2, "forcedReflowCount": 0.2, "longTaskCount": 0.2, "longAnimationFrameCount": 0.2 @@ -28,10 +27,10 @@ // Number of open→work→reset cycles "iterations": 3, - // Max acceptable total residual heap growth in MB. - // Each iteration cycles through ALL scenarios (text, code blocks, - // tool calls, thinking, terminal, multi-turn, etc.), so this needs - // to account for V8 internal caches that aren't immediately reclaimed. + // Max acceptable steady-state residual heap growth in MB, measured + // AFTER the first (warm-up) iteration. The first iteration's growth is + // dominated by one-time V8/JIT/module caches that aren't reclaimed and + // is excluded; a real leak keeps growing every subsequent iteration. "leakThresholdMB": 10 } } diff --git a/scripts/chat-simulation/merge-ci-summary.js b/scripts/chat-simulation/merge-ci-summary.js index 4c107f9f5dc..370291a0455 100644 --- a/scripts/chat-simulation/merge-ci-summary.js +++ b/scripts/chat-simulation/merge-ci-summary.js @@ -253,6 +253,7 @@ function generateUnifiedSummary(jsonReport, baseline, opts) { ['timeToFirstToken', 'timing', 'ms'], ['timeToComplete', 'timing', 'ms'], ['layoutCount', 'rendering', ''], + ['layoutDurationMs', 'rendering', 'ms'], ['recalcStyleCount', 'rendering', ''], ['forcedReflowCount', 'rendering', ''], ['longTaskCount', 'rendering', ''], @@ -267,8 +268,11 @@ function generateUnifiedSummary(jsonReport, baseline, opts) { ['extHostHeapDelta', 'extHost', 'MB'], ['extHostHeapDeltaPostGC', 'extHost', 'MB'], ]; + // layoutCount / recalcStyleCount are informational (inflated by CSS + // animations, compositor-driven, cheap) and do NOT gate — real layout cost is + // gated via layoutDurationMs below / timeToComplete. See SKILL.md. const regressionMetricNames = new Set([ - 'timeToFirstToken', 'timeToComplete', 'layoutCount', 'recalcStyleCount', + 'timeToFirstToken', 'timeToComplete', 'layoutDurationMs', 'forcedReflowCount', 'longTaskCount', 'longAnimationFrameCount', ]); diff --git a/scripts/chat-simulation/test-chat-mem-leaks.js b/scripts/chat-simulation/test-chat-mem-leaks.js index d63c4e4dcb4..945eeda1fef 100644 --- a/scripts/chat-simulation/test-chat-mem-leaks.js +++ b/scripts/chat-simulation/test-chat-mem-leaks.js @@ -25,7 +25,7 @@ * Usage: * npm run perf:chat-leak # defaults from config * npm run perf:chat-leak -- --iterations 5 # more iterations - * npm run perf:chat-leak -- --threshold 5 # 5MB total threshold + * npm run perf:chat-leak -- --threshold 5 # 5MB steady-state threshold * npm run perf:chat-leak -- --build 1.115.0 # test a specific build */ @@ -349,11 +349,24 @@ async function runLeakCheck(electronPath, mockServer, opts) { const totalResidualMB = Math.round((final.heapMB - baseline.heapMB) * 100) / 100; const totalResidualNodes = final.domNodes - baseline.domNodes; + // Steady-state residual EXCLUDES the first iteration. The first + // iteration's growth is dominated by one-time warm-up (V8 JIT, module + // and string caches, lazy singletons) rather than a leak — a real leak + // keeps growing every iteration, whereas warm-up plateaus. Basing the + // verdict on post-warm-up growth (heap relative to the end of iteration + // 1) avoids false positives from caching. Falls back to total residual + // when there are too few iterations to drop the warm-up one. + const warmupBaselineMB = iterationResults.length > 1 + ? iterationResults[0].afterHeapMB + : baseline.heapMB; + const steadyResidualMB = Math.round((final.heapMB - warmupBaselineMB) * 100) / 100; + return { baseline, final: { heapMB: final.heapMB, domNodes: final.domNodes }, totalResidualMB, totalResidualNodes, + steadyResidualMB, iterations: iterationResults, }; } finally { @@ -377,7 +390,7 @@ async function main() { registerPerfScenarios(); const mockServer = await startServer(0); - console.log(`[chat-simulation] Leak check: ${opts.iterations} iterations × ${getScenarioIds().length} scenarios, threshold ${opts.leakThresholdMB}MB total`); + console.log(`[chat-simulation] Leak check: ${opts.iterations} iterations × ${getScenarioIds().length} scenarios, threshold ${opts.leakThresholdMB}MB steady-state (excl. warm-up)`); console.log(`[chat-simulation] Build: ${electronPath}`); console.log(''); @@ -393,8 +406,9 @@ async function main() { console.log(` Iteration ${i + 1}: ${it.beforeHeapMB}MB → ${it.afterHeapMB}MB (residual: ${it.deltaHeapMB > 0 ? '+' : ''}${it.deltaHeapMB}MB, DOM: ${it.deltaDomNodes > 0 ? '+' : ''}${it.deltaDomNodes} nodes)`); } console.log(''); - console.log(` Total residual heap growth: ${result.totalResidualMB > 0 ? '+' : ''}${result.totalResidualMB}MB`); - console.log(` Total residual DOM growth: ${result.totalResidualNodes > 0 ? '+' : ''}${result.totalResidualNodes} nodes`); + console.log(` Total residual heap growth: ${result.totalResidualMB > 0 ? '+' : ''}${result.totalResidualMB}MB (includes one-time warm-up)`); + console.log(` Steady-state residual (excl. warm-up): ${result.steadyResidualMB > 0 ? '+' : ''}${result.steadyResidualMB}MB`); + console.log(` Total residual DOM growth: ${result.totalResidualNodes > 0 ? '+' : ''}${result.totalResidualNodes} nodes`); console.log(''); // Write JSON @@ -408,12 +422,12 @@ async function main() { }, null, 2)); console.log(`[chat-simulation] Results written to ${jsonPath}`); - const leaked = result.totalResidualMB > opts.leakThresholdMB; + const leaked = result.steadyResidualMB > opts.leakThresholdMB; console.log(''); if (leaked) { - console.log(`[chat-simulation] LEAK DETECTED — ${result.totalResidualMB}MB residual exceeds ${opts.leakThresholdMB}MB threshold`); + console.log(`[chat-simulation] LEAK DETECTED — ${result.steadyResidualMB}MB steady-state residual exceeds ${opts.leakThresholdMB}MB threshold`); } else { - console.log(`[chat-simulation] No leak detected (${result.totalResidualMB}MB residual < ${opts.leakThresholdMB}MB threshold)`); + console.log(`[chat-simulation] No leak detected (${result.steadyResidualMB}MB steady-state residual < ${opts.leakThresholdMB}MB threshold)`); } if (opts.ci) { @@ -429,11 +443,11 @@ async function main() { /** * Generate a Markdown summary for CI, matching the perf script pattern. - * @param {{ baseline: { heapMB: number, domNodes: number }, final: { heapMB: number, domNodes: number }, totalResidualMB: number, totalResidualNodes: number, iterations: { beforeHeapMB: number, afterHeapMB: number, deltaHeapMB: number, beforeDomNodes: number, afterDomNodes: number, deltaDomNodes: number }[] }} result + * @param {{ baseline: { heapMB: number, domNodes: number }, final: { heapMB: number, domNodes: number }, totalResidualMB: number, totalResidualNodes: number, steadyResidualMB: number, iterations: { beforeHeapMB: number, afterHeapMB: number, deltaHeapMB: number, beforeDomNodes: number, afterDomNodes: number, deltaDomNodes: number }[] }} result * @param {{ leakThresholdMB: number, iterations: number }} opts */ function generateLeakCISummary(result, opts) { - const leaked = result.totalResidualMB > opts.leakThresholdMB; + const leaked = result.steadyResidualMB > opts.leakThresholdMB; const verdict = leaked ? '\u274C **LEAK DETECTED**' : '\u2705 **No leak detected**'; const lines = []; lines.push('## Memory Leak Check'); @@ -441,7 +455,7 @@ function generateLeakCISummary(result, opts) { lines.push('| | |'); lines.push('|---|---|'); lines.push(`| **Verdict** | ${verdict} |`); - lines.push(`| **Threshold** | ${opts.leakThresholdMB} MB |`); + lines.push(`| **Threshold** | ${opts.leakThresholdMB} MB (steady-state, excl. warm-up) |`); lines.push(`| **Iterations** | ${opts.iterations} |`); lines.push(`| **Scenarios per iteration** | ${getScenarioIds().length} |`); lines.push(''); @@ -452,13 +466,17 @@ function generateLeakCISummary(result, opts) { const it = result.iterations[i]; const sign = it.deltaHeapMB > 0 ? '+' : ''; const domSign = it.deltaDomNodes > 0 ? '+' : ''; - lines.push(`| Iteration ${i + 1} | ${it.afterHeapMB} (${sign}${it.deltaHeapMB}) | ${it.afterDomNodes} (${domSign}${it.deltaDomNodes}) |`); + const note = i === 0 ? ' _(warm-up)_' : ''; + lines.push(`| Iteration ${i + 1}${note} | ${it.afterHeapMB} (${sign}${it.deltaHeapMB}) | ${it.afterDomNodes} (${domSign}${it.deltaDomNodes}) |`); } lines.push(`| **Final** | **${result.final.heapMB}** | **${result.final.domNodes}** |`); lines.push(''); + const steadySign = result.steadyResidualMB > 0 ? '+' : ''; const sign = result.totalResidualMB > 0 ? '+' : ''; const domSign = result.totalResidualNodes > 0 ? '+' : ''; - lines.push(`**Total residual growth:** ${sign}${result.totalResidualMB} MB heap, ${domSign}${result.totalResidualNodes} DOM nodes`); + lines.push(`**Steady-state residual (excl. warm-up):** ${steadySign}${result.steadyResidualMB} MB heap`); + lines.push(''); + lines.push(`**Total residual growth:** ${sign}${result.totalResidualMB} MB heap, ${domSign}${result.totalResidualNodes} DOM nodes _(includes one-time warm-up)_`); lines.push(''); return lines.join('\n'); } diff --git a/scripts/chat-simulation/test-chat-perf-regression.js b/scripts/chat-simulation/test-chat-perf-regression.js index 8dbf14a07b3..51228ff8e4a 100644 --- a/scripts/chat-simulation/test-chat-perf-regression.js +++ b/scripts/chat-simulation/test-chat-perf-regression.js @@ -47,6 +47,7 @@ function parseArgs() { noCache: false, force: false, heapSnapshots: false, + gcObjectStats: false, /** @type {string[]} */ scenarios: [], /** @type {string | undefined} */ @@ -100,6 +101,7 @@ function parseArgs() { case '--no-cache': opts.noCache = true; break; case '--force': opts.force = true; break; case '--heap-snapshots': opts.heapSnapshots = true; break; + case '--gc-object-stats': opts.gcObjectStats = true; break; case '--ci': opts.ci = true; opts.noCache = true; opts.heapSnapshots = true; opts.cleanupDiagnostics = true; break; case '--cleanup-diagnostics': opts.cleanupDiagnostics = true; break; case '--help': case '-h': @@ -128,6 +130,7 @@ function parseArgs() { ' --no-cache Ignore cached baseline data, always run fresh', ' --force Skip build mode mismatch confirmation', ' --heap-snapshots Take heap snapshots (slow; auto-enabled in --ci mode)', + ' --gc-object-stats Enable V8 gc_stats tracing for GC deep-dives only. WARNING: corrupts timings (adds ~550ms to any request hit by a major GC) — never use for benchmarking', ' --ci CI mode: write Markdown summary to ci-summary.md (implies --no-cache, --heap-snapshots, --cleanup-diagnostics)', ' --cleanup-diagnostics Remove heap snapshots, CPU profiles, and traces after each run to save disk space', ' --verbose Print per-run details', @@ -401,7 +404,7 @@ async function runOnce(electronPath, scenario, mockServer, verbose, runIndex, ru const extHostInspectPort = getNextExtHostInspectPort(); const vscode = await launchVSCode( electronPath, - buildArgs(userDataDir, extDir, logsDir, { isDevBuild, extHostInspectPort, traceFile: tracePath, appRoot }), + buildArgs(userDataDir, extDir, logsDir, { isDevBuild, extHostInspectPort, traceFile: tracePath, appRoot, gcObjectStats: runOpts?.gcObjectStats }), buildEnv(mockServer, { isDevBuild }), { verbose }, ); @@ -1003,6 +1006,7 @@ function generateCISummary(jsonReport, baseline, opts) { ['timeToFirstToken', 'timing', 'ms'], ['timeToComplete', 'timing', 'ms'], ['layoutCount', 'rendering', ''], + ['layoutDurationMs', 'rendering', 'ms'], ['recalcStyleCount', 'rendering', ''], ['forcedReflowCount', 'rendering', ''], ['longTaskCount', 'rendering', ''], @@ -1017,7 +1021,7 @@ function generateCISummary(jsonReport, baseline, opts) { ['extHostHeapDelta', 'extHost', 'MB'], ['extHostHeapDeltaPostGC', 'extHost', 'MB'], ]; - const regressionMetricNames = new Set(['timeToFirstToken', 'timeToComplete', 'forcedReflowCount', 'longTaskCount', 'longAnimationFrameCount']); + const regressionMetricNames = new Set(['timeToFirstToken', 'timeToComplete', 'layoutDurationMs', 'forcedReflowCount', 'longTaskCount', 'longAnimationFrameCount']); const lines = []; const scenarios = Object.keys(jsonReport.scenarios); @@ -1393,7 +1397,7 @@ async function main() { const runIdx = `${scenario}-resume-${prevTestRuns.length + i}`; console.log(`[chat-simulation] Run ${i + 1}/${runsToAdd}...`); try { - const m = await runOnce(testElectron, scenario, mockServer, opts.verbose, runIdx, prevDir, 'test', { ...opts.settingsOverrides, ...opts.testSettingsOverrides }, { heapSnapshots: opts.heapSnapshots }); + const m = await runOnce(testElectron, scenario, mockServer, opts.verbose, runIdx, prevDir, 'test', { ...opts.settingsOverrides, ...opts.testSettingsOverrides }, { heapSnapshots: opts.heapSnapshots, gcObjectStats: opts.gcObjectStats }); // Clean up previous run's diagnostics to bound disk usage; keep the latest if (opts.cleanupDiagnostics && prevTestRuns.length > 0) { cleanupRunDiagnostics(prevTestRuns[prevTestRuns.length - 1]); } prevTestRuns.push(m); @@ -1411,7 +1415,7 @@ async function main() { const runIdx = `baseline-${scenario}-resume-${prevBaseRuns.length + i}`; console.log(`[chat-simulation] Run ${i + 1}/${runsToAdd}...`); try { - const m = await runOnce(baselineElectron, scenario, mockServer, opts.verbose, runIdx, prevDir, 'baseline', { ...opts.settingsOverrides, ...opts.baselineSettingsOverrides }, { heapSnapshots: opts.heapSnapshots }); + const m = await runOnce(baselineElectron, scenario, mockServer, opts.verbose, runIdx, prevDir, 'baseline', { ...opts.settingsOverrides, ...opts.baselineSettingsOverrides }, { heapSnapshots: opts.heapSnapshots, gcObjectStats: opts.gcObjectStats }); // Clean up previous run's diagnostics to bound disk usage; keep the latest if (opts.cleanupDiagnostics && prevBaseRuns.length > 0) { cleanupRunDiagnostics(prevBaseRuns[prevBaseRuns.length - 1]); } prevBaseRuns.push(m); @@ -1557,7 +1561,7 @@ async function main() { const newResults = []; for (let i = 0; i < runsNeeded; i++) { try { - const m = await runOnce(baselineExePath, scenario, mockServer, opts.verbose, `baseline-${scenario}-${existingRuns.length + i}`, runDir, 'baseline', baselineSettings, { heapSnapshots: opts.heapSnapshots }); + const m = await runOnce(baselineExePath, scenario, mockServer, opts.verbose, `baseline-${scenario}-${existingRuns.length + i}`, runDir, 'baseline', baselineSettings, { heapSnapshots: opts.heapSnapshots, gcObjectStats: opts.gcObjectStats }); // Clean up previous run's diagnostics to bound disk usage; keep the latest if (opts.cleanupDiagnostics && newResults.length > 0) { cleanupRunDiagnostics(newResults[newResults.length - 1]); } newResults.push(m); @@ -1588,7 +1592,7 @@ async function main() { const results = []; for (let i = 0; i < opts.runs; i++) { try { - const m = await runOnce(baselineExePath, scenario, mockServer, opts.verbose, `baseline-${scenario}-${i}`, runDir, 'baseline', baselineSettings, { heapSnapshots: opts.heapSnapshots }); + const m = await runOnce(baselineExePath, scenario, mockServer, opts.verbose, `baseline-${scenario}-${i}`, runDir, 'baseline', baselineSettings, { heapSnapshots: opts.heapSnapshots, gcObjectStats: opts.gcObjectStats }); // Clean up previous run's diagnostics to bound disk usage; keep the latest if (opts.cleanupDiagnostics && results.length > 0) { cleanupRunDiagnostics(results[results.length - 1]); } results.push(m); @@ -1671,7 +1675,7 @@ async function main() { for (let i = 0; i < opts.runs; i++) { console.log(`[chat-simulation] Run ${i + 1}/${opts.runs}...`); try { - const metrics = await runOnce(electronPath, scenario, mockServer, opts.verbose, `${scenario}-${i}`, runDir, 'test', testSettings, { heapSnapshots: opts.heapSnapshots }); + const metrics = await runOnce(electronPath, scenario, mockServer, opts.verbose, `${scenario}-${i}`, runDir, 'test', testSettings, { heapSnapshots: opts.heapSnapshots, gcObjectStats: opts.gcObjectStats }); // Clean up previous run's diagnostics to bound disk usage; keep the latest if (opts.cleanupDiagnostics && results.length > 0) { cleanupRunDiagnostics(results[results.length - 1]); } results.push(metrics); @@ -1790,13 +1794,19 @@ async function printComparison(jsonReport, opts) { // [metric, group, unit] ['timeToFirstToken', 'timing', 'ms'], ['timeToComplete', 'timing', 'ms'], - ['layoutCount', 'rendering', ''], - ['recalcStyleCount', 'rendering', ''], + ['layoutDurationMs', 'rendering', 'ms'], ['forcedReflowCount', 'rendering', ''], ['longTaskCount', 'rendering', ''], ]; - // Informational metrics — shown in comparison but don't trigger failure + // Informational metrics — shown in comparison but don't trigger failure. + // layoutCount / recalcStyleCount are informational on purpose: they are + // inflated by CSS animations (compositor-driven, cheap) and don't reflect + // real cost — the real layout cost is layoutDurationMs (gated above). A + // build can do more, cheaper layouts yet spend less layout time and finish + // faster (e.g. giant-codeblock: +28% layoutCount but -7% layoutDurationMs). const infoMetrics = [ + ['layoutCount', 'rendering', ''], + ['recalcStyleCount', 'rendering', ''], ['heapDelta', 'memory', 'MB'], ['gcDurationMs', 'memory', 'ms'], ['extHostHeapDelta', 'extHost', 'MB'],