mirror of
https://github.com/microsoft/vscode.git
synced 2026-07-07 23:36:03 +01:00
Fix issues with chat perf pipeline (#323917)
This commit is contained in:
Paul
GitHub
@@ -42,6 +42,8 @@ npm run perf:chat-leak -- --messages 20 --verbose
|
||||
|
||||
Launches VS Code via Playwright Electron, opens the chat panel, sends a message with a mock LLM response, and measures timing, layout, and rendering metrics. By default, downloads VS Code 1.115.0 as a baseline, benchmarks it, then benchmarks the local dev build and compares.
|
||||
|
||||
> **You don't always need a baseline.** A baseline exists only for *comparison* (regression detection). If you just want the current build's numbers — profiling a single change, capturing traces/heap snapshots, or iterating on a scenario — pass `--no-baseline` to skip downloading and benchmarking the baseline entirely (roughly halves runtime). Baseline comparison is what turns raw measurements into a pass/fail verdict; without it you still get all the metrics, just no verdict.
|
||||
|
||||
### Key flags
|
||||
|
||||
| Flag | Default | Description |
|
||||
@@ -51,7 +53,7 @@ Launches VS Code via Playwright Electron, opens the chat panel, sends a message
|
||||
| `--build <path\|ver>` / `-b` | local dev | Build to test. Accepts path or version (`1.110.0`, `insiders`, commit hash). |
|
||||
| `--baseline <path>` | — | Compare against a previously saved baseline JSON file. |
|
||||
| `--baseline-build <path\|ver>` | `1.115.0` | Version or local path to benchmark as baseline. |
|
||||
| `--no-baseline` | — | Skip baseline comparison entirely. |
|
||||
| `--no-baseline` | — | Skip the baseline entirely — just measure the test build (no download, no comparison, ~2× faster). Use when you only need raw numbers, not a regression verdict. |
|
||||
| `--save-baseline` | — | Save results as the new baseline (requires `--baseline <path>`). |
|
||||
| `--resume <path>` | — | Resume a previous run, adding more iterations to increase confidence. |
|
||||
| `--threshold <frac>` | `0.2` | Regression threshold (0.2 = flag if 20% slower). |
|
||||
@@ -60,6 +62,7 @@ Launches VS Code via Playwright Electron, opens the chat panel, sends a message
|
||||
| `--force` | — | Skip build mode mismatch confirmation prompt. |
|
||||
| `--ci` | — | CI mode: write Markdown summary to `ci-summary.md` (implies `--no-cache`, `--heap-snapshots`, `--cleanup-diagnostics`). |
|
||||
| `--heap-snapshots` | — | Take heap snapshots after each run (slow; auto-enabled in `--ci` mode). |
|
||||
| `--gc-object-stats` | — | **GC deep-dives only.** Enables V8 `gc_stats` tracing (per-type heap object dump on every GC). ⚠️ Corrupts all timing metrics — a major GC landing mid-request adds ~550ms — so never use it for benchmarking. Off by default; prefer heap snapshots for memory analysis. |
|
||||
| `--cleanup-diagnostics` | — | Delete heap snapshots, CPU profiles, and traces to save disk. During runs, only the latest run's files are kept; after comparison, files for non-regressed scenarios are deleted. Auto-enabled in `--ci` mode. |
|
||||
| `--setting <k=v>` | — | Set a VS Code setting override for all builds (repeatable). |
|
||||
| `--test-setting <k=v>` | — | Set a VS Code setting override for the test build only. |
|
||||
@@ -186,13 +189,13 @@ Run `npm run perf:chat -- --help` to see the full list of registered scenario ID
|
||||
|
||||
Only these metrics trigger a regression failure (when they exceed the threshold with statistical significance):
|
||||
- `timeToFirstToken`, `timeToComplete` — user-perceived latency
|
||||
- `layoutDurationMs` — total layout time from the trace (the *real* layout cost)
|
||||
- `forcedReflowCount` — forced synchronous layouts are always bad
|
||||
- `longTaskCount`, `longAnimationFrameCount` — main thread jank
|
||||
|
||||
These are reported but **informational only** (won't fail CI):
|
||||
- `layoutCount` — inflated by CSS animations; use `layoutDurationMs` instead
|
||||
- `layoutDurationMs` — total layout time from trace (more meaningful than count)
|
||||
- `recalcStyleCount` — inflated by CSS animations (compositor-driven, cheap)
|
||||
- `layoutCount` — number of layout ops; inflated by CSS animations (compositor-driven, cheap). A build can do *more but cheaper* layouts, so gate on `layoutDurationMs`, not this count.
|
||||
- `recalcStyleCount` — number of style recalcs; inflated by CSS animations (compositor-driven, cheap)
|
||||
- `timeToRenderComplete` — includes typewriter animation tail
|
||||
- Memory/heap metrics — too noisy for single-request benchmarks
|
||||
|
||||
@@ -229,6 +232,54 @@ Launches one VS Code session, sends N messages sequentially, forces GC between e
|
||||
- DOM nodes stable after first message — normal (chat list virtualization working)
|
||||
- DOM nodes growing linearly — rendering leak, check disposable cleanup
|
||||
|
||||
## CI runs & pinpointing regressions
|
||||
|
||||
The perf + leak checks run in CI via the **`.github/workflows/chat-perf.yml`** workflow (a scheduled daily `workflow_dispatch`, plus manual dispatch). Each run benchmarks the current `main` as the **test** build against a fixed release **baseline** (from `config.jsonc`, e.g. `1.122.0`). Because the baseline is fixed, the **test** median for a metric across successive daily runs traces `main`'s trajectory over time — that's what lets you bisect *when* something regressed or went flaky.
|
||||
|
||||
### Finding and reading historic runs
|
||||
|
||||
```bash
|
||||
# List recent runs (most recent first) — note the run IDs and dates
|
||||
gh run list --workflow chat-perf.yml -R microsoft/vscode --limit 30 \
|
||||
--json databaseId,status,conclusion,createdAt,headBranch \
|
||||
--jq '.[] | [.databaseId, (.conclusion//.status), .createdAt, .headBranch] | @tsv'
|
||||
|
||||
# See a run's per-job results
|
||||
gh run view <run-id> -R microsoft/vscode --json jobs \
|
||||
--jq '.jobs[] | [.name, (.conclusion//.status)] | @tsv'
|
||||
|
||||
# Trigger a run manually against any ref/version:
|
||||
gh workflow run chat-perf.yml -R microsoft/vscode --ref main \
|
||||
-f test_build=<branch|sha|version> -f baseline_build=<version>
|
||||
```
|
||||
|
||||
### Artifacts per run (and retention — this matters for old runs)
|
||||
|
||||
| Artifact | Contents | Retention |
|
||||
|---|---|---|
|
||||
| `chat-perf-summary` | Unified `ci-summary.md`: verdicts, per-metric medians ±stddev, **per-run raw tables** | 30 days |
|
||||
| `perf-results-<group>` | Everything below **plus traces, CPU profiles, heap snapshots** (large) | 30 days |
|
||||
| `leak-results` | Leak log + `chat-simulation-leak-results.json` | 30 days |
|
||||
| `perf-summary-<group>` | `results.json` (full per-run metrics incl. `rawRuns`) + `baseline-*.json` | **1 day** |
|
||||
|
||||
```bash
|
||||
# List a run's artifacts + whether they've expired
|
||||
gh api repos/microsoft/vscode/actions/runs/<run-id>/artifacts \
|
||||
--jq '.artifacts[] | [.name, .expired] | @tsv'
|
||||
|
||||
# Download the human-readable summary (best first stop; survives 30 days)
|
||||
gh run download <run-id> -R microsoft/vscode -n chat-perf-summary
|
||||
```
|
||||
|
||||
### Pinpointing where a metric regressed / went flaky
|
||||
|
||||
1. **Bisect across dates.** Pull `chat-perf-summary/ci-summary.md` from several runs spanning the window. Compare the **test** median (and ±stddev) for the suspect metric+scenario run-to-run — the day it jumps is when `main` changed. A metric that goes *bimodal* (e.g. a raw-run column showing two clusters like `~250 / ~900`) is the flaky signature; a stable shift is a genuine regression.
|
||||
2. **Confirm it's real vs. measurement noise.** Check the per-run raw tables (in `ci-summary.md`) — high ±stddev / bimodal values mean the *median* is being pulled around by a few outlier runs, not a uniform slowdown.
|
||||
3. **Deep-dive the cause.** Download `perf-results-<group>` (has the `trace.json` per run) for a slow run and inspect what dominates the slow window. Trick that found the `gc_stats` artifact: sum main-thread `RunTask` durations between two `code/chat/*` marks (e.g. `willCollectInstructions` → `didCollectInstructions`) — if the window is ~0% busy, it's an async wait or a GC pause, not real work; then look at the largest `X`-phase events in that window (`MajorGC`, `Layout`, etc.).
|
||||
4. **Reproduce a suspect commit locally** to bisect precisely: `npm run perf:chat -- --build <sha> --baseline-build <ver> --runs 7` (or two commits directly via `--build <shaA> --baseline-build <shaB>`).
|
||||
|
||||
> Tip: `perf-summary-*` (the machine-readable `results.json` with `rawRuns`) is deleted after **1 day**, so for older runs rely on `chat-perf-summary` (raw tables, 30 days) or extract `results.json` from `perf-results-*` (also 30 days).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
|
||||
@@ -57,13 +57,19 @@ env:
|
||||
SCENARIOS_INPUT: ${{ inputs.scenarios || '' }}
|
||||
TEST_SETTINGS_INPUT: ${{ inputs.test_settings || '' }}
|
||||
BASELINE_SETTINGS_INPUT: ${{ inputs.baseline_settings || '' }}
|
||||
# Skip binary downloads during `npm ci` (they hang intermittently on hosted
|
||||
# runners inside the nested postinstall installs). Electron and Playwright are
|
||||
# fetched explicitly later via `Download Electron` and `Install Playwright
|
||||
# Chromium`, which ignore these flags. Mirrors pr-node-modules.yml.
|
||||
ELECTRON_SKIP_BINARY_DOWNLOAD: 1
|
||||
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: 1
|
||||
|
||||
jobs:
|
||||
# ── Shared setup: build once, cache everything ──────────────────────
|
||||
setup:
|
||||
name: Build & Cache
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
timeout-minutes: 50
|
||||
outputs:
|
||||
test_is_version: ${{ steps.resolve.outputs.is_version }}
|
||||
test_build_arg: ${{ steps.resolve.outputs.build_arg }}
|
||||
@@ -111,13 +117,33 @@ jobs:
|
||||
libasound2t64 libxshmfence1 libgtk-3-0
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
run: |
|
||||
set -e
|
||||
# Retry with a per-attempt timeout to survive transient npm registry
|
||||
# hangs/failures on hosted runners (mirrors pr-node-modules.yml).
|
||||
for i in {1..5}; do
|
||||
timeout 900 npm ci && break
|
||||
if [ $i -eq 5 ]; then
|
||||
echo "npm ci failed/hung too many times" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "npm ci attempt $i failed or timed out, retrying..."
|
||||
done
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Install build dependencies
|
||||
run: npm ci
|
||||
working-directory: build
|
||||
run: |
|
||||
set -e
|
||||
for i in {1..5}; do
|
||||
timeout 900 npm ci && break
|
||||
if [ $i -eq 5 ]; then
|
||||
echo "npm ci failed/hung too many times" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "npm ci attempt $i failed or timed out, retrying..."
|
||||
done
|
||||
|
||||
- name: Transpile source
|
||||
run: npm run transpile-client
|
||||
@@ -210,7 +236,16 @@ jobs:
|
||||
libasound2t64 libxshmfence1 libgtk-3-0
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
run: |
|
||||
set -e
|
||||
for i in {1..5}; do
|
||||
timeout 900 npm ci && break
|
||||
if [ $i -eq 5 ]; then
|
||||
echo "npm ci failed/hung too many times" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "npm ci attempt $i failed or timed out, retrying..."
|
||||
done
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
@@ -383,7 +418,16 @@ jobs:
|
||||
libasound2t64 libxshmfence1 libgtk-3-0
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
run: |
|
||||
set -e
|
||||
for i in {1..5}; do
|
||||
timeout 900 npm ci && break
|
||||
if [ $i -eq 5 ]; then
|
||||
echo "npm ci failed/hung too many times" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "npm ci attempt $i failed or timed out, retrying..."
|
||||
done
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
|
||||
@@ -217,12 +217,23 @@ function buildEnv(mockServer, { isDevBuild = true } = {}) {
|
||||
* @param {string} logsDir
|
||||
* @returns {string[]}
|
||||
*/
|
||||
function buildArgs(userDataDir, extDir, logsDir, { isDevBuild = true, extHostInspectPort = 0, traceFile = '', appRoot = ROOT } = {}) {
|
||||
function buildArgs(userDataDir, extDir, logsDir, { isDevBuild = true, extHostInspectPort = 0, traceFile = '', appRoot = ROOT, gcObjectStats = false } = {}) {
|
||||
// Chromium switches must come BEFORE the app path (ROOT) — Chromium
|
||||
// only processes switches that precede the first non-switch argument.
|
||||
const chromiumFlags = [];
|
||||
if (traceFile) {
|
||||
chromiumFlags.push(`--enable-tracing=v8.gc,disabled-by-default-v8.gc,disabled-by-default-v8.gc_stats,devtools.timeline,blink.user_timing`);
|
||||
// IMPORTANT: `disabled-by-default-v8.gc_stats` is intentionally OFF by
|
||||
// default. It makes V8 run GC_OBJECT_DUMP_STATISTICS (a full per-type
|
||||
// heap object dump) on every major GC, inflating a ~15ms GC pause to
|
||||
// ~550ms. When such a GC lands in the measured request window it
|
||||
// corrupts timeToFirstToken (bimodal ~250ms vs ~900ms). `v8.gc` +
|
||||
// `disabled-by-default-v8.gc` still provide the GC events we count.
|
||||
// Opt in via `--gc-object-stats` only for deliberate GC deep-dives
|
||||
// (never for timing runs), accepting that timings become unreliable.
|
||||
const gcCategories = gcObjectStats
|
||||
? 'v8.gc,disabled-by-default-v8.gc,disabled-by-default-v8.gc_stats'
|
||||
: 'v8.gc,disabled-by-default-v8.gc';
|
||||
chromiumFlags.push(`--enable-tracing=${gcCategories},devtools.timeline,blink.user_timing`);
|
||||
chromiumFlags.push(`--trace-startup-file=${traceFile}`);
|
||||
chromiumFlags.push(`--enable-tracing-format=json`);
|
||||
}
|
||||
|
||||
@@ -17,8 +17,7 @@
|
||||
"metricThresholds": {
|
||||
"timeToFirstToken": "100ms",
|
||||
"timeToComplete": 0.2,
|
||||
"layoutCount": 0.2,
|
||||
"recalcStyleCount": 0.2,
|
||||
"layoutDurationMs": 0.2,
|
||||
"forcedReflowCount": 0.2,
|
||||
"longTaskCount": 0.2,
|
||||
"longAnimationFrameCount": 0.2
|
||||
@@ -28,10 +27,10 @@
|
||||
// Number of open→work→reset cycles
|
||||
"iterations": 3,
|
||||
|
||||
// Max acceptable total residual heap growth in MB.
|
||||
// Each iteration cycles through ALL scenarios (text, code blocks,
|
||||
// tool calls, thinking, terminal, multi-turn, etc.), so this needs
|
||||
// to account for V8 internal caches that aren't immediately reclaimed.
|
||||
// Max acceptable steady-state residual heap growth in MB, measured
|
||||
// AFTER the first (warm-up) iteration. The first iteration's growth is
|
||||
// dominated by one-time V8/JIT/module caches that aren't reclaimed and
|
||||
// is excluded; a real leak keeps growing every subsequent iteration.
|
||||
"leakThresholdMB": 10
|
||||
}
|
||||
}
|
||||
|
||||
@@ -253,6 +253,7 @@ function generateUnifiedSummary(jsonReport, baseline, opts) {
|
||||
['timeToFirstToken', 'timing', 'ms'],
|
||||
['timeToComplete', 'timing', 'ms'],
|
||||
['layoutCount', 'rendering', ''],
|
||||
['layoutDurationMs', 'rendering', 'ms'],
|
||||
['recalcStyleCount', 'rendering', ''],
|
||||
['forcedReflowCount', 'rendering', ''],
|
||||
['longTaskCount', 'rendering', ''],
|
||||
@@ -267,8 +268,11 @@ function generateUnifiedSummary(jsonReport, baseline, opts) {
|
||||
['extHostHeapDelta', 'extHost', 'MB'],
|
||||
['extHostHeapDeltaPostGC', 'extHost', 'MB'],
|
||||
];
|
||||
// layoutCount / recalcStyleCount are informational (inflated by CSS
|
||||
// animations, compositor-driven, cheap) and do NOT gate — real layout cost is
|
||||
// gated via layoutDurationMs below / timeToComplete. See SKILL.md.
|
||||
const regressionMetricNames = new Set([
|
||||
'timeToFirstToken', 'timeToComplete', 'layoutCount', 'recalcStyleCount',
|
||||
'timeToFirstToken', 'timeToComplete', 'layoutDurationMs',
|
||||
'forcedReflowCount', 'longTaskCount', 'longAnimationFrameCount',
|
||||
]);
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@
|
||||
* Usage:
|
||||
* npm run perf:chat-leak # defaults from config
|
||||
* npm run perf:chat-leak -- --iterations 5 # more iterations
|
||||
* npm run perf:chat-leak -- --threshold 5 # 5MB total threshold
|
||||
* npm run perf:chat-leak -- --threshold 5 # 5MB steady-state threshold
|
||||
* npm run perf:chat-leak -- --build 1.115.0 # test a specific build
|
||||
*/
|
||||
|
||||
@@ -349,11 +349,24 @@ async function runLeakCheck(electronPath, mockServer, opts) {
|
||||
const totalResidualMB = Math.round((final.heapMB - baseline.heapMB) * 100) / 100;
|
||||
const totalResidualNodes = final.domNodes - baseline.domNodes;
|
||||
|
||||
// Steady-state residual EXCLUDES the first iteration. The first
|
||||
// iteration's growth is dominated by one-time warm-up (V8 JIT, module
|
||||
// and string caches, lazy singletons) rather than a leak — a real leak
|
||||
// keeps growing every iteration, whereas warm-up plateaus. Basing the
|
||||
// verdict on post-warm-up growth (heap relative to the end of iteration
|
||||
// 1) avoids false positives from caching. Falls back to total residual
|
||||
// when there are too few iterations to drop the warm-up one.
|
||||
const warmupBaselineMB = iterationResults.length > 1
|
||||
? iterationResults[0].afterHeapMB
|
||||
: baseline.heapMB;
|
||||
const steadyResidualMB = Math.round((final.heapMB - warmupBaselineMB) * 100) / 100;
|
||||
|
||||
return {
|
||||
baseline,
|
||||
final: { heapMB: final.heapMB, domNodes: final.domNodes },
|
||||
totalResidualMB,
|
||||
totalResidualNodes,
|
||||
steadyResidualMB,
|
||||
iterations: iterationResults,
|
||||
};
|
||||
} finally {
|
||||
@@ -377,7 +390,7 @@ async function main() {
|
||||
registerPerfScenarios();
|
||||
const mockServer = await startServer(0);
|
||||
|
||||
console.log(`[chat-simulation] Leak check: ${opts.iterations} iterations × ${getScenarioIds().length} scenarios, threshold ${opts.leakThresholdMB}MB total`);
|
||||
console.log(`[chat-simulation] Leak check: ${opts.iterations} iterations × ${getScenarioIds().length} scenarios, threshold ${opts.leakThresholdMB}MB steady-state (excl. warm-up)`);
|
||||
console.log(`[chat-simulation] Build: ${electronPath}`);
|
||||
console.log('');
|
||||
|
||||
@@ -393,8 +406,9 @@ async function main() {
|
||||
console.log(` Iteration ${i + 1}: ${it.beforeHeapMB}MB → ${it.afterHeapMB}MB (residual: ${it.deltaHeapMB > 0 ? '+' : ''}${it.deltaHeapMB}MB, DOM: ${it.deltaDomNodes > 0 ? '+' : ''}${it.deltaDomNodes} nodes)`);
|
||||
}
|
||||
console.log('');
|
||||
console.log(` Total residual heap growth: ${result.totalResidualMB > 0 ? '+' : ''}${result.totalResidualMB}MB`);
|
||||
console.log(` Total residual DOM growth: ${result.totalResidualNodes > 0 ? '+' : ''}${result.totalResidualNodes} nodes`);
|
||||
console.log(` Total residual heap growth: ${result.totalResidualMB > 0 ? '+' : ''}${result.totalResidualMB}MB (includes one-time warm-up)`);
|
||||
console.log(` Steady-state residual (excl. warm-up): ${result.steadyResidualMB > 0 ? '+' : ''}${result.steadyResidualMB}MB`);
|
||||
console.log(` Total residual DOM growth: ${result.totalResidualNodes > 0 ? '+' : ''}${result.totalResidualNodes} nodes`);
|
||||
console.log('');
|
||||
|
||||
// Write JSON
|
||||
@@ -408,12 +422,12 @@ async function main() {
|
||||
}, null, 2));
|
||||
console.log(`[chat-simulation] Results written to ${jsonPath}`);
|
||||
|
||||
const leaked = result.totalResidualMB > opts.leakThresholdMB;
|
||||
const leaked = result.steadyResidualMB > opts.leakThresholdMB;
|
||||
console.log('');
|
||||
if (leaked) {
|
||||
console.log(`[chat-simulation] LEAK DETECTED — ${result.totalResidualMB}MB residual exceeds ${opts.leakThresholdMB}MB threshold`);
|
||||
console.log(`[chat-simulation] LEAK DETECTED — ${result.steadyResidualMB}MB steady-state residual exceeds ${opts.leakThresholdMB}MB threshold`);
|
||||
} else {
|
||||
console.log(`[chat-simulation] No leak detected (${result.totalResidualMB}MB residual < ${opts.leakThresholdMB}MB threshold)`);
|
||||
console.log(`[chat-simulation] No leak detected (${result.steadyResidualMB}MB steady-state residual < ${opts.leakThresholdMB}MB threshold)`);
|
||||
}
|
||||
|
||||
if (opts.ci) {
|
||||
@@ -429,11 +443,11 @@ async function main() {
|
||||
|
||||
/**
|
||||
* Generate a Markdown summary for CI, matching the perf script pattern.
|
||||
* @param {{ baseline: { heapMB: number, domNodes: number }, final: { heapMB: number, domNodes: number }, totalResidualMB: number, totalResidualNodes: number, iterations: { beforeHeapMB: number, afterHeapMB: number, deltaHeapMB: number, beforeDomNodes: number, afterDomNodes: number, deltaDomNodes: number }[] }} result
|
||||
* @param {{ baseline: { heapMB: number, domNodes: number }, final: { heapMB: number, domNodes: number }, totalResidualMB: number, totalResidualNodes: number, steadyResidualMB: number, iterations: { beforeHeapMB: number, afterHeapMB: number, deltaHeapMB: number, beforeDomNodes: number, afterDomNodes: number, deltaDomNodes: number }[] }} result
|
||||
* @param {{ leakThresholdMB: number, iterations: number }} opts
|
||||
*/
|
||||
function generateLeakCISummary(result, opts) {
|
||||
const leaked = result.totalResidualMB > opts.leakThresholdMB;
|
||||
const leaked = result.steadyResidualMB > opts.leakThresholdMB;
|
||||
const verdict = leaked ? '\u274C **LEAK DETECTED**' : '\u2705 **No leak detected**';
|
||||
const lines = [];
|
||||
lines.push('## Memory Leak Check');
|
||||
@@ -441,7 +455,7 @@ function generateLeakCISummary(result, opts) {
|
||||
lines.push('| | |');
|
||||
lines.push('|---|---|');
|
||||
lines.push(`| **Verdict** | ${verdict} |`);
|
||||
lines.push(`| **Threshold** | ${opts.leakThresholdMB} MB |`);
|
||||
lines.push(`| **Threshold** | ${opts.leakThresholdMB} MB (steady-state, excl. warm-up) |`);
|
||||
lines.push(`| **Iterations** | ${opts.iterations} |`);
|
||||
lines.push(`| **Scenarios per iteration** | ${getScenarioIds().length} |`);
|
||||
lines.push('');
|
||||
@@ -452,13 +466,17 @@ function generateLeakCISummary(result, opts) {
|
||||
const it = result.iterations[i];
|
||||
const sign = it.deltaHeapMB > 0 ? '+' : '';
|
||||
const domSign = it.deltaDomNodes > 0 ? '+' : '';
|
||||
lines.push(`| Iteration ${i + 1} | ${it.afterHeapMB} (${sign}${it.deltaHeapMB}) | ${it.afterDomNodes} (${domSign}${it.deltaDomNodes}) |`);
|
||||
const note = i === 0 ? ' _(warm-up)_' : '';
|
||||
lines.push(`| Iteration ${i + 1}${note} | ${it.afterHeapMB} (${sign}${it.deltaHeapMB}) | ${it.afterDomNodes} (${domSign}${it.deltaDomNodes}) |`);
|
||||
}
|
||||
lines.push(`| **Final** | **${result.final.heapMB}** | **${result.final.domNodes}** |`);
|
||||
lines.push('');
|
||||
const steadySign = result.steadyResidualMB > 0 ? '+' : '';
|
||||
const sign = result.totalResidualMB > 0 ? '+' : '';
|
||||
const domSign = result.totalResidualNodes > 0 ? '+' : '';
|
||||
lines.push(`**Total residual growth:** ${sign}${result.totalResidualMB} MB heap, ${domSign}${result.totalResidualNodes} DOM nodes`);
|
||||
lines.push(`**Steady-state residual (excl. warm-up):** ${steadySign}${result.steadyResidualMB} MB heap`);
|
||||
lines.push('');
|
||||
lines.push(`**Total residual growth:** ${sign}${result.totalResidualMB} MB heap, ${domSign}${result.totalResidualNodes} DOM nodes _(includes one-time warm-up)_`);
|
||||
lines.push('');
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
@@ -47,6 +47,7 @@ function parseArgs() {
|
||||
noCache: false,
|
||||
force: false,
|
||||
heapSnapshots: false,
|
||||
gcObjectStats: false,
|
||||
/** @type {string[]} */
|
||||
scenarios: [],
|
||||
/** @type {string | undefined} */
|
||||
@@ -100,6 +101,7 @@ function parseArgs() {
|
||||
case '--no-cache': opts.noCache = true; break;
|
||||
case '--force': opts.force = true; break;
|
||||
case '--heap-snapshots': opts.heapSnapshots = true; break;
|
||||
case '--gc-object-stats': opts.gcObjectStats = true; break;
|
||||
case '--ci': opts.ci = true; opts.noCache = true; opts.heapSnapshots = true; opts.cleanupDiagnostics = true; break;
|
||||
case '--cleanup-diagnostics': opts.cleanupDiagnostics = true; break;
|
||||
case '--help': case '-h':
|
||||
@@ -128,6 +130,7 @@ function parseArgs() {
|
||||
' --no-cache Ignore cached baseline data, always run fresh',
|
||||
' --force Skip build mode mismatch confirmation',
|
||||
' --heap-snapshots Take heap snapshots (slow; auto-enabled in --ci mode)',
|
||||
' --gc-object-stats Enable V8 gc_stats tracing for GC deep-dives only. WARNING: corrupts timings (adds ~550ms to any request hit by a major GC) — never use for benchmarking',
|
||||
' --ci CI mode: write Markdown summary to ci-summary.md (implies --no-cache, --heap-snapshots, --cleanup-diagnostics)',
|
||||
' --cleanup-diagnostics Remove heap snapshots, CPU profiles, and traces after each run to save disk space',
|
||||
' --verbose Print per-run details',
|
||||
@@ -401,7 +404,7 @@ async function runOnce(electronPath, scenario, mockServer, verbose, runIndex, ru
|
||||
const extHostInspectPort = getNextExtHostInspectPort();
|
||||
const vscode = await launchVSCode(
|
||||
electronPath,
|
||||
buildArgs(userDataDir, extDir, logsDir, { isDevBuild, extHostInspectPort, traceFile: tracePath, appRoot }),
|
||||
buildArgs(userDataDir, extDir, logsDir, { isDevBuild, extHostInspectPort, traceFile: tracePath, appRoot, gcObjectStats: runOpts?.gcObjectStats }),
|
||||
buildEnv(mockServer, { isDevBuild }),
|
||||
{ verbose },
|
||||
);
|
||||
@@ -1003,6 +1006,7 @@ function generateCISummary(jsonReport, baseline, opts) {
|
||||
['timeToFirstToken', 'timing', 'ms'],
|
||||
['timeToComplete', 'timing', 'ms'],
|
||||
['layoutCount', 'rendering', ''],
|
||||
['layoutDurationMs', 'rendering', 'ms'],
|
||||
['recalcStyleCount', 'rendering', ''],
|
||||
['forcedReflowCount', 'rendering', ''],
|
||||
['longTaskCount', 'rendering', ''],
|
||||
@@ -1017,7 +1021,7 @@ function generateCISummary(jsonReport, baseline, opts) {
|
||||
['extHostHeapDelta', 'extHost', 'MB'],
|
||||
['extHostHeapDeltaPostGC', 'extHost', 'MB'],
|
||||
];
|
||||
const regressionMetricNames = new Set(['timeToFirstToken', 'timeToComplete', 'forcedReflowCount', 'longTaskCount', 'longAnimationFrameCount']);
|
||||
const regressionMetricNames = new Set(['timeToFirstToken', 'timeToComplete', 'layoutDurationMs', 'forcedReflowCount', 'longTaskCount', 'longAnimationFrameCount']);
|
||||
|
||||
const lines = [];
|
||||
const scenarios = Object.keys(jsonReport.scenarios);
|
||||
@@ -1393,7 +1397,7 @@ async function main() {
|
||||
const runIdx = `${scenario}-resume-${prevTestRuns.length + i}`;
|
||||
console.log(`[chat-simulation] Run ${i + 1}/${runsToAdd}...`);
|
||||
try {
|
||||
const m = await runOnce(testElectron, scenario, mockServer, opts.verbose, runIdx, prevDir, 'test', { ...opts.settingsOverrides, ...opts.testSettingsOverrides }, { heapSnapshots: opts.heapSnapshots });
|
||||
const m = await runOnce(testElectron, scenario, mockServer, opts.verbose, runIdx, prevDir, 'test', { ...opts.settingsOverrides, ...opts.testSettingsOverrides }, { heapSnapshots: opts.heapSnapshots, gcObjectStats: opts.gcObjectStats });
|
||||
// Clean up previous run's diagnostics to bound disk usage; keep the latest
|
||||
if (opts.cleanupDiagnostics && prevTestRuns.length > 0) { cleanupRunDiagnostics(prevTestRuns[prevTestRuns.length - 1]); }
|
||||
prevTestRuns.push(m);
|
||||
@@ -1411,7 +1415,7 @@ async function main() {
|
||||
const runIdx = `baseline-${scenario}-resume-${prevBaseRuns.length + i}`;
|
||||
console.log(`[chat-simulation] Run ${i + 1}/${runsToAdd}...`);
|
||||
try {
|
||||
const m = await runOnce(baselineElectron, scenario, mockServer, opts.verbose, runIdx, prevDir, 'baseline', { ...opts.settingsOverrides, ...opts.baselineSettingsOverrides }, { heapSnapshots: opts.heapSnapshots });
|
||||
const m = await runOnce(baselineElectron, scenario, mockServer, opts.verbose, runIdx, prevDir, 'baseline', { ...opts.settingsOverrides, ...opts.baselineSettingsOverrides }, { heapSnapshots: opts.heapSnapshots, gcObjectStats: opts.gcObjectStats });
|
||||
// Clean up previous run's diagnostics to bound disk usage; keep the latest
|
||||
if (opts.cleanupDiagnostics && prevBaseRuns.length > 0) { cleanupRunDiagnostics(prevBaseRuns[prevBaseRuns.length - 1]); }
|
||||
prevBaseRuns.push(m);
|
||||
@@ -1557,7 +1561,7 @@ async function main() {
|
||||
const newResults = [];
|
||||
for (let i = 0; i < runsNeeded; i++) {
|
||||
try {
|
||||
const m = await runOnce(baselineExePath, scenario, mockServer, opts.verbose, `baseline-${scenario}-${existingRuns.length + i}`, runDir, 'baseline', baselineSettings, { heapSnapshots: opts.heapSnapshots });
|
||||
const m = await runOnce(baselineExePath, scenario, mockServer, opts.verbose, `baseline-${scenario}-${existingRuns.length + i}`, runDir, 'baseline', baselineSettings, { heapSnapshots: opts.heapSnapshots, gcObjectStats: opts.gcObjectStats });
|
||||
// Clean up previous run's diagnostics to bound disk usage; keep the latest
|
||||
if (opts.cleanupDiagnostics && newResults.length > 0) { cleanupRunDiagnostics(newResults[newResults.length - 1]); }
|
||||
newResults.push(m);
|
||||
@@ -1588,7 +1592,7 @@ async function main() {
|
||||
const results = [];
|
||||
for (let i = 0; i < opts.runs; i++) {
|
||||
try {
|
||||
const m = await runOnce(baselineExePath, scenario, mockServer, opts.verbose, `baseline-${scenario}-${i}`, runDir, 'baseline', baselineSettings, { heapSnapshots: opts.heapSnapshots });
|
||||
const m = await runOnce(baselineExePath, scenario, mockServer, opts.verbose, `baseline-${scenario}-${i}`, runDir, 'baseline', baselineSettings, { heapSnapshots: opts.heapSnapshots, gcObjectStats: opts.gcObjectStats });
|
||||
// Clean up previous run's diagnostics to bound disk usage; keep the latest
|
||||
if (opts.cleanupDiagnostics && results.length > 0) { cleanupRunDiagnostics(results[results.length - 1]); }
|
||||
results.push(m);
|
||||
@@ -1671,7 +1675,7 @@ async function main() {
|
||||
for (let i = 0; i < opts.runs; i++) {
|
||||
console.log(`[chat-simulation] Run ${i + 1}/${opts.runs}...`);
|
||||
try {
|
||||
const metrics = await runOnce(electronPath, scenario, mockServer, opts.verbose, `${scenario}-${i}`, runDir, 'test', testSettings, { heapSnapshots: opts.heapSnapshots });
|
||||
const metrics = await runOnce(electronPath, scenario, mockServer, opts.verbose, `${scenario}-${i}`, runDir, 'test', testSettings, { heapSnapshots: opts.heapSnapshots, gcObjectStats: opts.gcObjectStats });
|
||||
// Clean up previous run's diagnostics to bound disk usage; keep the latest
|
||||
if (opts.cleanupDiagnostics && results.length > 0) { cleanupRunDiagnostics(results[results.length - 1]); }
|
||||
results.push(metrics);
|
||||
@@ -1790,13 +1794,19 @@ async function printComparison(jsonReport, opts) {
|
||||
// [metric, group, unit]
|
||||
['timeToFirstToken', 'timing', 'ms'],
|
||||
['timeToComplete', 'timing', 'ms'],
|
||||
['layoutCount', 'rendering', ''],
|
||||
['recalcStyleCount', 'rendering', ''],
|
||||
['layoutDurationMs', 'rendering', 'ms'],
|
||||
['forcedReflowCount', 'rendering', ''],
|
||||
['longTaskCount', 'rendering', ''],
|
||||
];
|
||||
// Informational metrics — shown in comparison but don't trigger failure
|
||||
// Informational metrics — shown in comparison but don't trigger failure.
|
||||
// layoutCount / recalcStyleCount are informational on purpose: they are
|
||||
// inflated by CSS animations (compositor-driven, cheap) and don't reflect
|
||||
// real cost — the real layout cost is layoutDurationMs (gated above). A
|
||||
// build can do more, cheaper layouts yet spend less layout time and finish
|
||||
// faster (e.g. giant-codeblock: +28% layoutCount but -7% layoutDurationMs).
|
||||
const infoMetrics = [
|
||||
['layoutCount', 'rendering', ''],
|
||||
['recalcStyleCount', 'rendering', ''],
|
||||
['heapDelta', 'memory', 'MB'],
|
||||
['gcDurationMs', 'memory', 'ms'],
|
||||
['extHostHeapDelta', 'extHost', 'MB'],
|
||||
|
||||
Reference in New Issue
Block a user