mirror of
https://github.com/microsoft/vscode.git
synced 2026-07-10 16:49:36 +01:00
574c85547f
* use the existing editTracker to help emit edit survival metrics, tracked at full file level for now * attempt to grade survival per chunk, extracting chunk locations from the tool call parameters * build the file n-gram set once per chunked-survival call Previously computeChunkedFourGramSurvival called computeFractionPresentIn for each chunk, which rebuilt the file's 4-gram Set every time. Cost was O(|chunks| * |file|); a single chunked-scoring call against a 500 KB file with 50 chunks did ~25 MB of repeated Set builds, and that ran for every one of the 7 samples in the 15-min schedule. Factor out buildNGramSet + computeFractionPresentInSet and have the chunked path build the set exactly once. Adds a regression test that asserts 50 chunks against a 500 KB file scores in <1s. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * skip edit-survival tracking for files larger than 5 MB Reporter samples the file 7 times over 15 minutes and pins beforeText / afterText snapshots for the duration. Past a certain size the per-sample n-gram work and snapshot retention stop paying for themselves -- the file is almost certainly not something the AI meaningfully authored end-to-end, and survival math on it would be dominated by churn we don't care about. 5 MB still covers every realistic source file; chosen as a single threshold for both whole-file and chunked scoring modes because noRevert (the dominant per-sample cost) is identical between them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * only treat FILE_NOT_FOUND as deletion in edit-survival reporter Transient readFile errors (permissions, FILE_TOO_LARGE, provider hiccups, etc.) were being recorded as 'file deleted' events, which both biased telemetry toward false reverts and stopped sampling early. Now we only mark the file as deleted when the underlying FileOperationResult is FILE_NOT_FOUND; other errors log a warning, skip the sample, and the reporter keeps running for subsequent samples. Addresses PR review feedback on #321233. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * move AI-chunk extraction into FileEditTracker The agent sessions (Claude observer + Copilot session) were each calling extractAiChunks() before takeCompletedEdit() and passing the result through. The tracker is the only consumer of the chunks (it forwards them to the survival reporter), so callers don't need to know about that detail. Move the extraction inside the tracker; callers now just pass the toolName + raw tool input they already have. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * tidy comments and document chunked-scoring coverage invariant - Drop stale historical references (Phase 8 prefix, 'production extension' pointers, 'previously / atm / deliberately' wording) and reword to describe current behavior only. - Soften the .ipynb skip comments to note we may revisit once we have a notebook-aware tracker, rather than ruling notebooks out. - Document in editChunkExtractor.ts that every tool the agent host currently treats as a file-edit tool has a matching case here, so the whole-file scoring path is a safety net for SDK shape drift / newly added tools rather than a regular code path. Cross-reference from IEditSurvivalReporterLaunchParams.aiChunks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix tsgo override signature in RecordingTelemetryService NullTelemetryServiceShape.publicLog2 declares zero parameters; tsgo (stricter than tsc on override compatibility) rejects an override that adds required-shaped parameters. Match the base by declaring the override with optional params, and drop the now-unused gdprTypings imports. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * add modelId and aiCharCount to edit-survival telemetry Stamps each agentHost.trackEditSurvival event with the model that produced the edit and the AI-written character count, so dashboards can slice by model and weight survival by AI contribution (matching the chat ext's ghcopilot_committed_characters_hll_model_sku rollup). Claude reads the model per assistant message, which is naturally correct for subagents. Copilot tracks the session's last-seen model via setModel and onUsage -- best-effort for subagents but accurate for the parent agent. aiCharCount is 0 in the whole-file fallback (coverage invariant means this filters out ~0% of real traffic; dashboards should filter scoringMode='chunked'). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * warn when edit-survival modelId is missing modelId should always be populated in practice. An empty modelId in the emitted telemetry now indicates a bug worth investigating, and the tracker logs a warning at edit-completion time so we notice locally too. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * doc: correct modelId subagent-attribution claim The Copilot CLI SDK's Usage event fires per LLM call with the model that actually ran it -- including subagent calls. So _lastSeenModelId correctly attributes subagent edits, not just parent-agent edits. Verified against a live sonnet-parent / haiku-subagent run where the subagent's three tool calls correctly emitted modelId=claude-haiku-4.5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * doc: trim _lastSeenModelId comment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
124 lines
5.7 KiB
TypeScript
124 lines
5.7 KiB
TypeScript
/*---------------------------------------------------------------------------------------------
|
|
* Copyright (c) Microsoft Corporation. All rights reserved.
|
|
* Licensed under the MIT License. See License.txt in the project root for license information.
|
|
*--------------------------------------------------------------------------------------------*/
|
|
|
|
import type { SDKMessage } from '@anthropic-ai/claude-agent-sdk';
|
|
|
|
import assert from 'assert';
|
|
import { VSBuffer } from '../../../../base/common/buffer.js';
|
|
import { IReference } from '../../../../base/common/lifecycle.js';
|
|
import { URI } from '../../../../base/common/uri.js';
|
|
import { ensureNoDisposablesAreLeakedInTestSuite } from '../../../../base/test/common/utils.js';
|
|
import { FileService } from '../../../files/common/fileService.js';
|
|
import { IFileService } from '../../../files/common/files.js';
|
|
import { InMemoryFileSystemProvider } from '../../../files/common/inMemoryFilesystemProvider.js';
|
|
import { IInstantiationService } from '../../../instantiation/common/instantiation.js';
|
|
import { InstantiationService } from '../../../instantiation/common/instantiationService.js';
|
|
import { ServiceCollection } from '../../../instantiation/common/serviceCollection.js';
|
|
import { ILogService, NullLogService } from '../../../log/common/log.js';
|
|
import { IDiffComputeService } from '../../common/diffComputeService.js';
|
|
import { ISessionDatabase } from '../../common/sessionDataService.js';
|
|
import { ToolResultContentType } from '../../common/state/sessionState.js';
|
|
import { ClaudeFileEditObserver } from '../../node/claude/claudeFileEditObserver.js';
|
|
import { ClaudeMapperState } from '../../node/claude/claudeMapSessionEvents.js';
|
|
import { IEditSurvivalReporterFactory, NullEditSurvivalReporterFactory } from '../../node/shared/editSurvivalReporter.js';
|
|
import { createZeroDiffComputeService, TestSessionDatabase } from '../common/sessionTestHelpers.js';
|
|
|
|
interface IObserverHarness {
|
|
readonly observer: ClaudeFileEditObserver;
|
|
readonly db: TestSessionDatabase;
|
|
readonly fileService: FileService;
|
|
readonly mapperState: ClaudeMapperState;
|
|
}
|
|
|
|
function createObserver(disposables: Pick<import('../../../../base/common/lifecycle.js').DisposableStore, 'add'>): IObserverHarness {
|
|
const fileService = disposables.add(new FileService(new NullLogService()));
|
|
const fs = disposables.add(new InMemoryFileSystemProvider());
|
|
disposables.add(fileService.registerProvider('file', fs));
|
|
|
|
const db = new TestSessionDatabase();
|
|
const dbRef: IReference<ISessionDatabase> = { object: db, dispose: () => { } };
|
|
|
|
const services = new ServiceCollection(
|
|
[ILogService, new NullLogService()],
|
|
[IFileService, fileService],
|
|
[IDiffComputeService, createZeroDiffComputeService()],
|
|
[IEditSurvivalReporterFactory, new NullEditSurvivalReporterFactory()],
|
|
);
|
|
const instantiationService: IInstantiationService = disposables.add(new InstantiationService(services));
|
|
const observer = disposables.add(instantiationService.createInstance(
|
|
ClaudeFileEditObserver,
|
|
'claude:/sess-1',
|
|
dbRef,
|
|
));
|
|
return { observer, db, fileService, mapperState: new ClaudeMapperState() };
|
|
}
|
|
|
|
function assistantMessage(content: unknown): Extract<SDKMessage, { type: 'assistant' }> {
|
|
return { type: 'assistant', message: { content } } as Extract<SDKMessage, { type: 'assistant' }>;
|
|
}
|
|
|
|
function userMessage(content: unknown): Extract<SDKMessage, { type: 'user' }> {
|
|
return { type: 'user', message: { content } } as Extract<SDKMessage, { type: 'user' }>;
|
|
}
|
|
|
|
suite('ClaudeFileEditObserver', () => {
|
|
|
|
const disposables = ensureNoDisposablesAreLeakedInTestSuite();
|
|
|
|
test('observe assistant→user round-trip caches a file edit on the mapper state', async () => {
|
|
const { observer, fileService, mapperState } = createObserver(disposables);
|
|
await fileService.writeFile(URI.file('/work/a.txt'), VSBuffer.fromString('before'));
|
|
|
|
observer.observeAssistant(assistantMessage([
|
|
{ type: 'tool_use', id: 'tu-1', name: 'Write', input: { file_path: '/work/a.txt', content: 'after' } },
|
|
]));
|
|
|
|
// Tool runs (we simulate it here): file content changes.
|
|
await fileService.writeFile(URI.file('/work/a.txt'), VSBuffer.fromString('after'));
|
|
|
|
await observer.observeUser(userMessage([
|
|
{ type: 'tool_result', tool_use_id: 'tu-1', content: 'ok' },
|
|
]), 'turn-1', mapperState);
|
|
|
|
const cached = mapperState.takeFileEdit('tu-1');
|
|
assert.ok(cached, 'expected a cached file edit');
|
|
assert.strictEqual(cached.type, ToolResultContentType.FileEdit);
|
|
});
|
|
|
|
test('observeAssistant ignores non-edit tools and tools with no path', () => {
|
|
const { observer, mapperState } = createObserver(disposables);
|
|
|
|
observer.observeAssistant(assistantMessage([
|
|
{ type: 'tool_use', id: 'tu-bash', name: 'Bash', input: { command: 'ls' } },
|
|
{ type: 'tool_use', id: 'tu-bad', name: 'Write', input: {} }, // missing file_path
|
|
{ type: 'text', text: 'hi' },
|
|
]));
|
|
|
|
// No tool_result for either id should produce a cached edit.
|
|
assert.strictEqual(mapperState.takeFileEdit('tu-bash'), undefined);
|
|
assert.strictEqual(mapperState.takeFileEdit('tu-bad'), undefined);
|
|
});
|
|
|
|
test('observeUser ignores tool_result for unknown tool_use_id', async () => {
|
|
const { observer, mapperState } = createObserver(disposables);
|
|
|
|
await observer.observeUser(userMessage([
|
|
{ type: 'tool_result', tool_use_id: 'never-tracked', content: 'ok' },
|
|
]), 'turn-1', mapperState);
|
|
|
|
assert.strictEqual(mapperState.takeFileEdit('never-tracked'), undefined);
|
|
});
|
|
|
|
test('observe* tolerate non-array message content', async () => {
|
|
const { observer, mapperState } = createObserver(disposables);
|
|
|
|
observer.observeAssistant(assistantMessage('plain string body'));
|
|
await observer.observeUser(userMessage('plain string body'), 'turn-1', mapperState);
|
|
|
|
// No throws, no cached edits.
|
|
assert.strictEqual(mapperState.takeFileEdit('any'), undefined);
|
|
});
|
|
});
|