Files
vscode/src/vs/platform/agentHost/test/node/claudeFileEditObserver.test.ts
T
Aaron Munger 574c85547f agent host edit survival tracking (#321233)
* use the existing editTracker to help emit edit survival metrics, tracked at full file level for now

* attempt to grade survival per chunk, extracting chunk locations from the tool call parameters

* build the file n-gram set once per chunked-survival call

Previously computeChunkedFourGramSurvival called computeFractionPresentIn
for each chunk, which rebuilt the file's 4-gram Set every time. Cost
was O(|chunks| * |file|); a single chunked-scoring call against a
500 KB file with 50 chunks did ~25 MB of repeated Set builds, and
that ran for every one of the 7 samples in the 15-min schedule.

Factor out buildNGramSet + computeFractionPresentInSet and have the
chunked path build the set exactly once. Adds a regression test that
asserts 50 chunks against a 500 KB file scores in <1s.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* skip edit-survival tracking for files larger than 5 MB

Reporter samples the file 7 times over 15 minutes and pins
beforeText / afterText snapshots for the duration. Past a certain
size the per-sample n-gram work and snapshot retention stop paying
for themselves -- the file is almost certainly not something the AI
meaningfully authored end-to-end, and survival math on it would be
dominated by churn we don't care about.

5 MB still covers every realistic source file; chosen as a single
threshold for both whole-file and chunked scoring modes because
noRevert (the dominant per-sample cost) is identical between them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* only treat FILE_NOT_FOUND as deletion in edit-survival reporter

Transient readFile errors (permissions, FILE_TOO_LARGE, provider
hiccups, etc.) were being recorded as 'file deleted' events, which
both biased telemetry toward false reverts and stopped sampling
early. Now we only mark the file as deleted when the underlying
FileOperationResult is FILE_NOT_FOUND; other errors log a warning,
skip the sample, and the reporter keeps running for subsequent
samples.

Addresses PR review feedback on #321233.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* move AI-chunk extraction into FileEditTracker

The agent sessions (Claude observer + Copilot session) were each
calling extractAiChunks() before takeCompletedEdit() and passing the
result through. The tracker is the only consumer of the chunks
(it forwards them to the survival reporter), so callers don't need
to know about that detail. Move the extraction inside the tracker;
callers now just pass the toolName + raw tool input they already
have.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* tidy comments and document chunked-scoring coverage invariant

- Drop stale historical references (Phase 8 prefix, 'production
  extension' pointers, 'previously / atm / deliberately' wording) and
  reword to describe current behavior only.
- Soften the .ipynb skip comments to note we may revisit once we have
  a notebook-aware tracker, rather than ruling notebooks out.
- Document in editChunkExtractor.ts that every tool the agent host
  currently treats as a file-edit tool has a matching case here, so
  the whole-file scoring path is a safety net for SDK shape drift /
  newly added tools rather than a regular code path. Cross-reference
  from IEditSurvivalReporterLaunchParams.aiChunks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix tsgo override signature in RecordingTelemetryService

NullTelemetryServiceShape.publicLog2 declares zero parameters; tsgo
(stricter than tsc on override compatibility) rejects an override
that adds required-shaped parameters. Match the base by declaring
the override with optional params, and drop the now-unused
gdprTypings imports.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* add modelId and aiCharCount to edit-survival telemetry

Stamps each agentHost.trackEditSurvival event with the model that produced
the edit and the AI-written character count, so dashboards can slice by
model and weight survival by AI contribution (matching the chat ext's
ghcopilot_committed_characters_hll_model_sku rollup).

Claude reads the model per assistant message, which is naturally correct
for subagents. Copilot tracks the session's last-seen model via setModel
and onUsage -- best-effort for subagents but accurate for the parent agent.

aiCharCount is 0 in the whole-file fallback (coverage invariant means this
filters out ~0% of real traffic; dashboards should filter scoringMode='chunked').

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* warn when edit-survival modelId is missing

modelId should always be populated in practice. An empty modelId in the
emitted telemetry now indicates a bug worth investigating, and the tracker
logs a warning at edit-completion time so we notice locally too.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* doc: correct modelId subagent-attribution claim

The Copilot CLI SDK's Usage event fires per LLM call with the model that
actually ran it -- including subagent calls. So _lastSeenModelId correctly
attributes subagent edits, not just parent-agent edits. Verified against a
live sonnet-parent / haiku-subagent run where the subagent's three tool
calls correctly emitted modelId=claude-haiku-4.5.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* doc: trim _lastSeenModelId comment

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-06-17 14:24:28 +10:00

124 lines
5.7 KiB
TypeScript

/*---------------------------------------------------------------------------------------------
* Copyright (c) Microsoft Corporation. All rights reserved.
* Licensed under the MIT License. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/
import type { SDKMessage } from '@anthropic-ai/claude-agent-sdk';
import assert from 'assert';
import { VSBuffer } from '../../../../base/common/buffer.js';
import { IReference } from '../../../../base/common/lifecycle.js';
import { URI } from '../../../../base/common/uri.js';
import { ensureNoDisposablesAreLeakedInTestSuite } from '../../../../base/test/common/utils.js';
import { FileService } from '../../../files/common/fileService.js';
import { IFileService } from '../../../files/common/files.js';
import { InMemoryFileSystemProvider } from '../../../files/common/inMemoryFilesystemProvider.js';
import { IInstantiationService } from '../../../instantiation/common/instantiation.js';
import { InstantiationService } from '../../../instantiation/common/instantiationService.js';
import { ServiceCollection } from '../../../instantiation/common/serviceCollection.js';
import { ILogService, NullLogService } from '../../../log/common/log.js';
import { IDiffComputeService } from '../../common/diffComputeService.js';
import { ISessionDatabase } from '../../common/sessionDataService.js';
import { ToolResultContentType } from '../../common/state/sessionState.js';
import { ClaudeFileEditObserver } from '../../node/claude/claudeFileEditObserver.js';
import { ClaudeMapperState } from '../../node/claude/claudeMapSessionEvents.js';
import { IEditSurvivalReporterFactory, NullEditSurvivalReporterFactory } from '../../node/shared/editSurvivalReporter.js';
import { createZeroDiffComputeService, TestSessionDatabase } from '../common/sessionTestHelpers.js';
interface IObserverHarness {
readonly observer: ClaudeFileEditObserver;
readonly db: TestSessionDatabase;
readonly fileService: FileService;
readonly mapperState: ClaudeMapperState;
}
function createObserver(disposables: Pick<import('../../../../base/common/lifecycle.js').DisposableStore, 'add'>): IObserverHarness {
const fileService = disposables.add(new FileService(new NullLogService()));
const fs = disposables.add(new InMemoryFileSystemProvider());
disposables.add(fileService.registerProvider('file', fs));
const db = new TestSessionDatabase();
const dbRef: IReference<ISessionDatabase> = { object: db, dispose: () => { } };
const services = new ServiceCollection(
[ILogService, new NullLogService()],
[IFileService, fileService],
[IDiffComputeService, createZeroDiffComputeService()],
[IEditSurvivalReporterFactory, new NullEditSurvivalReporterFactory()],
);
const instantiationService: IInstantiationService = disposables.add(new InstantiationService(services));
const observer = disposables.add(instantiationService.createInstance(
ClaudeFileEditObserver,
'claude:/sess-1',
dbRef,
));
return { observer, db, fileService, mapperState: new ClaudeMapperState() };
}
function assistantMessage(content: unknown): Extract<SDKMessage, { type: 'assistant' }> {
return { type: 'assistant', message: { content } } as Extract<SDKMessage, { type: 'assistant' }>;
}
function userMessage(content: unknown): Extract<SDKMessage, { type: 'user' }> {
return { type: 'user', message: { content } } as Extract<SDKMessage, { type: 'user' }>;
}
suite('ClaudeFileEditObserver', () => {
const disposables = ensureNoDisposablesAreLeakedInTestSuite();
test('observe assistant→user round-trip caches a file edit on the mapper state', async () => {
const { observer, fileService, mapperState } = createObserver(disposables);
await fileService.writeFile(URI.file('/work/a.txt'), VSBuffer.fromString('before'));
observer.observeAssistant(assistantMessage([
{ type: 'tool_use', id: 'tu-1', name: 'Write', input: { file_path: '/work/a.txt', content: 'after' } },
]));
// Tool runs (we simulate it here): file content changes.
await fileService.writeFile(URI.file('/work/a.txt'), VSBuffer.fromString('after'));
await observer.observeUser(userMessage([
{ type: 'tool_result', tool_use_id: 'tu-1', content: 'ok' },
]), 'turn-1', mapperState);
const cached = mapperState.takeFileEdit('tu-1');
assert.ok(cached, 'expected a cached file edit');
assert.strictEqual(cached.type, ToolResultContentType.FileEdit);
});
test('observeAssistant ignores non-edit tools and tools with no path', () => {
const { observer, mapperState } = createObserver(disposables);
observer.observeAssistant(assistantMessage([
{ type: 'tool_use', id: 'tu-bash', name: 'Bash', input: { command: 'ls' } },
{ type: 'tool_use', id: 'tu-bad', name: 'Write', input: {} }, // missing file_path
{ type: 'text', text: 'hi' },
]));
// No tool_result for either id should produce a cached edit.
assert.strictEqual(mapperState.takeFileEdit('tu-bash'), undefined);
assert.strictEqual(mapperState.takeFileEdit('tu-bad'), undefined);
});
test('observeUser ignores tool_result for unknown tool_use_id', async () => {
const { observer, mapperState } = createObserver(disposables);
await observer.observeUser(userMessage([
{ type: 'tool_result', tool_use_id: 'never-tracked', content: 'ok' },
]), 'turn-1', mapperState);
assert.strictEqual(mapperState.takeFileEdit('never-tracked'), undefined);
});
test('observe* tolerate non-array message content', async () => {
const { observer, mapperState } = createObserver(disposables);
observer.observeAssistant(assistantMessage('plain string body'));
await observer.observeUser(userMessage('plain string body'), 'turn-1', mapperState);
// No throws, no cached edits.
assert.strictEqual(mapperState.takeFileEdit('any'), undefined);
});
});