mirror of
https://github.com/microsoft/vscode.git
synced 2026-08-15 10:15:04 +01:00
Add symbolicate-crash-dump skill (#325168)
* Add symbolicate-crash-dump skill Adds a project skill that walks through symbolicating a native VS Code crash dump (.dmp) with electron-minidump, including how to attach Electron/Insiders/Stable symbol files, how to read the resulting backtrace to find the crashing module and process, and the gdb-based flow for remote extension host core dumps. Distilled from the 'Symbolicating a crash dump' section of the Native-Crash-Issues wiki page. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Correct symbol-download guidance after real-world verification Verified the manual symbol-attach flow against a shipped Stable dump (VS Code 1.128.0, Electron 42.5.0, win32-x64): - Symbol zips are keyed by exact breakpad debug-id; a same-version zip from a different build has a non-matching id and is not used. - The *-symbols-*.zip files only carry a few first-party modules (electron.exe, libEGL, libGLESv2); runtime.node is not published and often cannot be symbolicated. - On Windows the shipped Code.exe maps to electron.exe symbols; document the <module>.pdb/<debug-id>/<module>.sym layout and hash matching. - Add a fallback note: when no matching-hash symbols exist, attribute by module and process rather than expecting method names on every frame. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Document authenticated download from private prebuilt repo microsoft/vscode-electron-prebuilt is private, so a plain link/curl 404s. Add a gh-authenticated 'gh release download' snippet, note the gh-auth prerequisite, and clarify Code-OSS symbols come from the public electron/electron releases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: fix tilde expansion and rename table header - Move the tilde outside quotes in the cp example so it expands to the home directory instead of being treated as a literal path segment. - Rename the 'Signal' column to 'Marker' to avoid confusion with OS signals in a crash-analysis context. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Tyler James Leonhardt
co-authored by
Copilot
Copilot
parent
1e9d77576d
commit
1bd2ae02eb
@@ -0,0 +1,180 @@
|
||||
---
|
||||
name: symbolicate-crash-dump
|
||||
description: "Symbolicate a native VS Code crash dump (.dmp) using electron-minidump. Use when given a crash dump file, asked to symbolicate a crash, resolve missing method names in a native crash backtrace, or attach Electron/Insiders/Stable symbol files. VS Code team members only; requires macOS or Linux."
|
||||
---
|
||||
|
||||
# Symbolicate a Crash Dump
|
||||
|
||||
Turn a native VS Code crash dump (`.dmp`) into a readable backtrace with method names using [electron-minidump](https://www.npmjs.com/package/electron-minidump).
|
||||
|
||||
> **VS Code team members only.** Symbol files for internal Electron, Insiders, and Stable builds live in a private-adjacent release repo. A **macOS or Linux** device is required — electron-minidump does not run on Windows.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A crash dump file (`*.dmp`). See [Creating a crash report](#creating-a-crash-report) below if you don't have one yet.
|
||||
- A global install of `electron-minidump`:
|
||||
```bash
|
||||
npm install -g electron-minidump
|
||||
```
|
||||
- For Insiders/Stable symbols, an authenticated GitHub CLI (`gh auth status`) with access to the private `microsoft/vscode-electron-prebuilt` repo.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Run an initial symbolication pass
|
||||
|
||||
This generates or refreshes the electron-minidump cache and tells you which symbols are still missing.
|
||||
|
||||
```bash
|
||||
electron-minidump crash-file.dmp > symbolicated-output.log
|
||||
```
|
||||
|
||||
Inspect `symbolicated-output.log`. Look at the top frames of the backtrace: if a frame names a module (e.g. `Electron Framework`) but has **no method name after it**, symbols for that module are required.
|
||||
|
||||
> **Retry on transient download errors.** electron-minidump downloads symbols from public symbol servers, so a run can fail on a transient network error (e.g. `failed to download ... (code 56)` or `(code 28)`). Successfully downloaded symbols are cached, so simply re-running the same command resumes where it left off. Retrying a few times is expected:
|
||||
> ```bash
|
||||
> for i in 1 2 3 4 5; do
|
||||
> electron-minidump crash-file.dmp > symbolicated-output.log && break
|
||||
> sleep 3
|
||||
> done
|
||||
> ```
|
||||
|
||||
### 2. Get the appropriate symbol files
|
||||
|
||||
Match the symbol source to the build that produced the crash:
|
||||
|
||||
| Build that crashed | Symbol files source |
|
||||
|--------------------|---------------------|
|
||||
| Insiders / Stable (internal Electron) | [microsoft/vscode-electron-prebuilt releases](https://github.com/microsoft/vscode-electron-prebuilt/releases) |
|
||||
| Code - OSS (OSS Electron) | [electron/electron releases](https://github.com/electron/electron/releases) |
|
||||
|
||||
`microsoft/vscode-electron-prebuilt` is a **private** repo — this is why the flow is team-members-only. A plain browser or `curl` link will 404 without auth; download the asset with an authenticated GitHub CLI instead (`gh auth status` should show you logged in):
|
||||
|
||||
```bash
|
||||
# List releases (tagged by Electron version) to find the right tag:
|
||||
gh release list --repo microsoft/vscode-electron-prebuilt
|
||||
|
||||
# Download just the symbol zip you need:
|
||||
gh release download v42.5.0-14525058 \
|
||||
--repo microsoft/vscode-electron-prebuilt \
|
||||
--pattern "stable-symbols-v42.5.0-win32-x64.zip"
|
||||
```
|
||||
|
||||
The releases are tagged by **Electron version**, not VS Code version, so first find the Electron version the crashed VS Code build shipped. It's the `target=` in that version's `.npmrc` (e.g. `git show 1.128.0:.npmrc`), which mirrors the `electron` devDependency in `package.json`. Then pick the matching symbol zip by **quality, platform, and architecture** — e.g. a Stable Windows x64 crash on Electron 42.5.0 needs `stable-symbols-v42.5.0-win32-x64.zip` (use `insiders-symbols-…` for Insiders). Code - OSS symbols come from the public [electron/electron releases](https://github.com/electron/electron/releases) and can be downloaded without special access.
|
||||
|
||||
> **These zips are small and selective.** A `*-symbols-*.zip` typically contains only a handful of first-party modules — `electron.exe.sym`, `libEGL.dll.sym`, `libGLESv2.dll.sym` on Windows (and the equivalents elsewhere). Many modules that show up in a backtrace — notably `runtime.node` and any OS/third-party DLL — are **not** in these zips. `runtime.node` frames often cannot be symbolicated at all from public symbols; when the crash is in a third-party module, attribute it by module name rather than expecting method names on every frame (see [Reading the result](#reading-the-result)).
|
||||
|
||||
### 3. Copy the `.sym` files into the electron-minidump cache
|
||||
|
||||
The cache lives at:
|
||||
|
||||
```bash
|
||||
"$(npm root -g)/electron-minidump/cache/breakpad_symbols"
|
||||
```
|
||||
|
||||
Breakpad keys symbols by `<module>.pdb/<debug-id>/<module>.sym`, and the `<debug-id>` **must match exactly** between the dump and the symbol zip — a same-version zip built from a different pipeline run will have a different id and won't be used. After the initial pass, the cache already contains an (empty) directory for each required module whose name and hash you must match:
|
||||
|
||||
```bash
|
||||
CACHE="$(npm root -g)/electron-minidump/cache/breakpad_symbols"
|
||||
# The debug-id electron-minidump wants for a given module:
|
||||
ls "$CACHE/electron.exe.pdb" # e.g. DD081533CD7E33A44C4C44205044422E1
|
||||
```
|
||||
|
||||
Unzip the downloaded symbols and confirm the same module/hash exists in the zip before copying it in. The zip's internal layout is also `<module>.pdb/<debug-id>/<module>.sym`.
|
||||
|
||||
Example (Windows: the shipped main binary is `Code.exe`, but its symbols come from `electron.exe.sym`):
|
||||
|
||||
```bash
|
||||
# Unzip somewhere, e.g. ~/stable-symbols/
|
||||
unzip stable-symbols-v42.5.0-win32-x64.zip -d ~/stable-symbols
|
||||
|
||||
# Only copy if the debug-id matches what the cache expects.
|
||||
# Keep the tilde outside quotes so it expands to your home directory:
|
||||
HASH=DD081533CD7E33A44C4C44205044422E1
|
||||
cp ~/stable-symbols/symbols/electron.exe.pdb/"$HASH"/electron.exe.sym \
|
||||
"$CACHE/electron.exe.pdb/$HASH/"
|
||||
```
|
||||
|
||||
On macOS the analogous module is `Electron Framework` (`Electron Framework/<hash>/Electron Framework.sym`). Repeat for any other module the initial pass reported as missing method names — but only when the zip actually contains a matching-hash `.sym` for it.
|
||||
|
||||
> **If no matching-hash symbols exist**, you cannot get method names for that module — this is common for officially-shipped Stable/Insiders builds whose exact `electron.exe`/`Code.exe` hash isn't in any public prebuilt zip, and for `runtime.node`. Fall back to attributing the crash by module and process (see [Reading the result](#reading-the-result)); that is usually enough to identify a third-party culprit.
|
||||
|
||||
### 4. Re-run symbolication
|
||||
|
||||
```bash
|
||||
electron-minidump crash-file.dmp > symbolicated-output.log
|
||||
```
|
||||
|
||||
The backtrace in `symbolicated-output.log` should now have method names attached. If some frames are still bare, return to step 2 for the module(s) still missing symbols.
|
||||
|
||||
## Reading the result
|
||||
|
||||
Once you have a symbolicated backtrace, turn it into a root cause by answering two questions:
|
||||
|
||||
### Which module owns the crash?
|
||||
|
||||
Look at the **top frame of the crashing thread** (marked `(crashed)`) and its module name:
|
||||
|
||||
- If it's a **VS Code / Electron module** — `Code.exe`, `runtime.node`, `Electron Framework`, `libnode`, `libffmpeg`, V8 frames — the fault is likely inside the product or Electron.
|
||||
- If it's a **third-party / OS module** — an antivirus, VPN, proxy, or shell-extension DLL injected into the process — the crash is almost certainly caused by that software, not VS Code. Injected DLLs often appear interleaved with `runtime.node`/V8 frames because they hook the runtime.
|
||||
|
||||
Find where the module is loaded on disk to confirm it's third-party. On Windows the `strings` of the dump usually reveal the full path, e.g. a DLL under `C:\WINDOWS\system32\` or a vendor folder rather than the VS Code install directory:
|
||||
|
||||
```bash
|
||||
strings -a crash-file.dmp | grep -i "<SuspectModule>" | sort -u
|
||||
```
|
||||
|
||||
### Which process crashed?
|
||||
|
||||
The process type tells you whether this is the main process, a renderer/window, or the extension host. On modern Electron the **extension host runs inside a Node utility process**, so a crash there shows up as the `node.mojom.NodeService` utility — not a process literally named "extension host":
|
||||
|
||||
```bash
|
||||
strings -a crash-file.dmp | grep -oiE "utility-sub-type=[a-zA-Z0-9._-]+|node\.mojom\.[A-Za-z]+|--type=[a-z-]+" | sort -u
|
||||
```
|
||||
|
||||
| Marker | Process |
|
||||
|--------|---------|
|
||||
| `node.mojom.NodeService` / `utility-sub-type=node.mojom.NodeService` | Extension host (Node utility process) |
|
||||
| `--type=renderer` | A workbench window (renderer) |
|
||||
| `--type=gpu-process` | GPU process |
|
||||
| (no `--type`) | Main process |
|
||||
|
||||
Match this against the reported symptom (e.g. an "extension host crash-loop" should correspond to a `NodeService` utility crash).
|
||||
|
||||
When you have multiple dumps, symbolicate each and compare the crash reason and top frame — an identical signature across dumps confirms a single, reproducible cause.
|
||||
|
||||
## Creating a crash report
|
||||
|
||||
If you don't yet have a `.dmp` file, produce one with the `--crash-reporter-directory` option:
|
||||
|
||||
1. Close all instances of VS Code.
|
||||
2. Run `code --crash-reporter-directory <absolute-path>` from the command line (use `code-insiders` for the Insiders build).
|
||||
3. Take the steps that lead to the crash.
|
||||
4. Look for a `*.dmp` file in that folder.
|
||||
|
||||
You can only symbolicate crashes from a build you have matching symbols for. Crashes from Insiders/Stable need the internal Electron symbols; crashes from a local source build (Code - OSS) need the OSS Electron symbols.
|
||||
|
||||
## Remote Extension Host crashes (Linux, gdb)
|
||||
|
||||
Native crashes in a remote server's extension host use core dumps and `gdb` instead of electron-minidump:
|
||||
|
||||
1. Before running the server, allow core dumps: `ulimit -c unlimited`.
|
||||
2. Reproduce the crash. Retrieve the core dump via `coredumpctl`, or from the path in `/proc/sys/kernel/core_pattern`.
|
||||
3. Load it in gdb and capture output:
|
||||
```bash
|
||||
gdb -se <path-to-vscode-server>/node -c <path-to-core-file>
|
||||
```
|
||||
Then run and collect the output of:
|
||||
```
|
||||
set pagination off
|
||||
info sharedlibrary
|
||||
info registers
|
||||
bt full
|
||||
disassemble
|
||||
```
|
||||
|
||||
## Tips
|
||||
|
||||
- Keep `crash-file.dmp` and `symbolicated-output.log` out of the repo — they are throwaway artifacts.
|
||||
- The cache path is dynamic; always resolve it with `$(npm root -g)` rather than hardcoding a home directory.
|
||||
- Breakpad matches symbols by exact debug-id, not by version name. If method names are still missing after adding symbols, confirm the `.sym` you copied has the exact hash the cache directory expects, and double-check the quality (`stable`/`insiders`), platform (`win32`/`darwin`/`linux`), and arch (`x64`/`arm64`) of the symbol zip.
|
||||
- Not every frame can be symbolicated. `runtime.node` and third-party/OS modules frequently have no public symbols; identifying the crashing module and process is usually enough to reach a root cause.
|
||||
Reference in New Issue
Block a user