Commit Graph
546 Commits
Author SHA1 Message Date
joshspicer 3ae3ff0ded Add file path setup for mock policy server and improve state polling (#333272)
Agent Host changes for agents/mock-policy-server-file-paths-setup
2026-08-29 00:06:39 +00:00
joshspicerandCopilot d9b5c14fd7 policy: add "sandbox, no internet" preset to mock-policy-server (#333258)
Adds a managed-settings preset that enables the agent runtime sandbox with bypass allowed while denying outbound network access, so sandboxed tools run offline. Makes it easy to exercise the no-internet sandbox policy path against the local mock server.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-08-28 15:06:58 -07:00
Dmitriy VasyuraandCopilot 002b200b89 Reuse cached Electron for unit tests (#333137)
Route the unit-test launchers through the version-aware Electron preparation path so repeated runs avoid re-downloading and re-extracting the same runtime.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-08-28 12:20:16 -07:00
Ben Villalobos 445946b860 agentHost: remove stale AHP sync workaround (#333048)
* agentHost: remove stale AHP sync workaround

The pinned AHP revision already removes the unused automation action import. Restore verbatim sync behavior and the default sibling repository lookup.

* agentHost: retain AHP source override

* signing commit
2026-08-27 17:10:52 -07:00
33e6a5d6f3 automations: feat: migrate execution to Agent Host Protocol (#331796)
* automations: feat: migrate execution to Agent Host Protocol

Move Automation definitions, scheduling, run lifecycle, and persistence into the Agent Host while safely migrating legacy VS Code data and retaining older-host fallback behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 142bb750-abf2-4b29-91b8-1e9ab2444635

* automations: guard Agent Host migration cutover

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* automations: optimize sparse cron evaluation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* automations: gate initial Run advertisement on enabled state

Match the canRun composite used by handleConfigurationChanged so a create that arrives while chat.automations.enabled is false does not advertise Run.

* automations/ahp: gate SessionWorkingDirectoryReplaced action

Bring Replaced to parity with Set/Removed at the working-directory gate:
extend the action union, canonicalize both URIs in the resolver, enforce
editor-only client and provider capability at _dispatchActionNow, and
include Replaced in the customization-enablement listener.

Also honors the Removed contract for primaryReplacement: rejects index-0
removal when the agent advertises primaryReplacement.

* automations/ahp: enforce single-run invariant in schedule claim loop

Break out of the trigger loop once a schedule trigger has been claimed
for an Automation, after advancing that trigger's cursor. Prevents two
simultaneously-due schedule triggers on one Automation from both starting
sessions and violating the one-non-terminal-run-per-Automation invariant.
Deferred triggers keep their cursors untouched so their firings are
re-evaluated on the next tick rather than dropped.

* automations/ahp: coalesce simultaneously-due schedule triggers into one run

Two schedule triggers on one Automation whose past-due cursors land in
the same claim tick now coalesce into a single run. Catch-up is
idempotent: one run at now, regardless of how many missed firings a
sibling trigger also carries. The claim block skips when another trigger
has already claimed for this Automation this tick, but the deferred
cursor still rolls forward to its next cron occurrence so it does not
re-fire on the next tick. Replaces the earlier break-after-claim
approach from e2c2657, which serialized the deferred firing back-to-back.

* automations/ahp: gate Run authority on legacy import until source is durably removed

The migration path published imported snapshots with Run granted before the legacy source row was CAS-removed, creating a double-authority window where both schedulers could dispatch the same occurrence. If the removal failed, the window became permanent.

Stage imports with a pending meta flag, centralize the Run/Remove permission check in the host, restore Remove when the flag clears, gate scheduling ownership on the flag, and add an acknowledge hook so cross-provider retargets clear the pending state after the source is durably gone. Recovery drains stranded pending rows on reconnect.

* automations: finish Agent Host merge integration

Co-authored-by: benvillalobos <4691428+benvillalobos@users.noreply.github.com>

* test: mirror host automation migration authority

Co-authored-by: benvillalobos <4691428+benvillalobos@users.noreply.github.com>

* agentHost: restore main's reject of SessionWorkingDirectoryReplaced

The merge collapsed main's two-block structure for working-directory actions back into one, dropping the explicit reject for `session/workingDirectoryReplaced`. No provider advertises `primaryReplacement` and the host has no backend side effect for the action, so the reducer would apply an unvalidated mutation. Restore the standalone reject before the EditorWindow-gated block for Set / Removed.

* signing commit

* signing commit

* automations: use family guards for dispatch, matching existing pattern

Recreate the pre-existing dispatch-guard convention rather than switching
this hot path to isClientDispatchable. The generic check pulled in synced
protocol code and widened the scope of this change. Automation and
automation-run actions now flow through family guards, consistent with how
session, chat, terminal, changeset, and annotations actions are already
handled. The family-vs-permission gap this restores is pre-existing and
tracked for maintainer follow-up.

* automations: drop reducer-helpers sync patch, no longer needed

The dispatch guard no longer uses isClientDispatchable, so nothing imports
the synced reducer-helpers.ts. Its generated-source compatibility patch only
existed to widen that helper's signature for the guard, so remove it and let
the file sync verbatim. The state.ts dead-import patch stays until the synced
AHP revision picks up the upstream fix.

* agentHost: separate subscription resources and channels

Keep URI-based subscription APIs narrow while preserving exact AHP catalogue channels. Mark failed reconnect restorations by channel and cover the exact-channel path with regressions.

* automations: give the catalogue channel a round-trippable authority

The catalogue channel constant was `ahp-automations://`, which is not a
round-trippable URI. `URI.parse('ahp-automations://').toString()` drops the
empty authority and yields `ahp-automations:`, so a channel serialized on the
client no longer matched the catalogue check on the host.

Append a `catalog` authority so the URI survives a parse/toString round-trip.
Comparing catalogue channels as URIs everywhere (ResourceMap/isEqual) remains
the intended followup.

* automations: key the catalogue channel like every other channel

Now that the catalogue channel URI round-trips through parse/toString, its
subscription key no longer needs to preserve the raw channel string. Drop the
`_subscriptionChannel` helper and its automation-catalogue special case, and
key every channel through `_subscriptionResource` by its parsed URI.

Comparing channels as URIs everywhere (ResourceMap/isEqual) remains the
intended followup.

* automations: use shared action family routing

Use dedicated automation channels for subscription relevance and reuse the canonical action-family guards in state management. This avoids silent drift when the protocol adds actions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8c6d543f-7777-4e8b-9371-3e39a0842293

* automations: inject the automation service as a collaborator

AgentService received the automation service through a post-construction setAutomationService setter, leaving a definite-assignment field and a one-off wiring step. It depends on AgentService only through the lazy callback adapter, so it can be built first and passed in the collaborators bag like every other dependency. Its constructor installs durable state without firing emitters, so the earlier ordering is safe.

* automations: key subscriptions by URI through ResourceMap

The subscription map had been changed from a ResourceMap to a Map<string> keyed by a synthetic getComparisonKey string, purely to hold the lossy catalogue channel under a raw key. With the catalogue channel now round-trippable and the special-case keying gone, every entry keys by a real URI again. Restores the ResourceMap that main uses and drops the synthetic key from the entry type, the resource helper, and all fifteen call sites.

* automations: revert subscribe callbacks to URI

The subscription manager threaded raw channel strings through _subscribe/_unsubscribe to dodge a lossy round-trip on the catalogue channel. Now that the catalogue URI round-trips, revert those callbacks to (resource: URI) to match main. The wire boundary keeps its .toString() serialization in the protocol client.

* automations: migrate legacy definitions to native AHP state

Translate legacy automations at the client boundary instead of persisting editor projection metadata. Derive the compatibility view from host state and canonicalize supported round trips.

* automations: stabilize legacy target serialization for AHP migration

Serialize folderUri as explicit URI components instead of URI.toJSON().
toJSON() only emits the lazily cached fsPath and formatted fields once
they have been accessed, so two URIs for the same folder could serialize
differently. That made the snapshot equality check during Agent Host
migration fail with "kept changing while migrating" for every
folder-target automation, blocking migration indefinitely.

Reads already go through URI.revive, so existing ledger data stays
compatible in both directions.

* automations: ignore rejected chat actions when finalizing runs

_handleEnvelope finalized an automation run on ChatTurnComplete,
ChatTurnCancelled, or ChatError but did not check rejectionReason. A
rejected action never reached authoritative host state, so applying it
marked a still-live run terminal and orphaned its session. Guard against
rejected envelopes before finalizing, matching the sessions provider's
action handler.

* automations: salvage valid legacy ledger entries

Keep valid automations writable when individual persisted rows are malformed. Update migration coverage and compare round-tripped URI resources without relying on cache state.

* automations: recover corrupt legacy run archives

Salvage valid archived runs and repair unreadable current-version archives during import. Preserve fail-closed handling for unsupported newer versions.

* automations: wait for provider migration before wakeup

Refresh pending automations only after initial provider migration succeeds. Apply the same ordering when a failed provider migration is retried.

* automations: reject inactive catalog subscriptions

Apply the standard subscription cancellation guard before adding an automation catalogue subscriber.

* automations: surface pending import drain failures

* automations: acknowledge migrated snapshots

* signing commit

---------

Co-authored-by: Ben Villalobos <4691428+benvillalobos@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Ben Villalobos <bevillal@microsoft.com>
Copilot-Session: 142bb750-abf2-4b29-91b8-1e9ab2444635
Copilot-Session: 8c6d543f-7777-4e8b-9371-3e39a0842293
2026-08-27 14:29:23 -07:00
joshspicerandCopilot 84ef3481c6 mock policy server: improve cross-platform setup (#332833)
Make proxy and SDK cache setup explicit for macOS and Windows, and let developers navigate live requests directly to their endpoint configuration. Preserve keyboard focus while the request log refreshes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-08-27 05:50:05 +00:00
joshspicerandCopilot 8ae285809c managed settings: forceRemoteSettingsRefresh fails closed (#332388)
* policy: add managed-settings freshness contract

Groundwork for making `forceRemoteSettingsRefresh` a real fail-closed
startup gate (microsoft/vscode-internalbacklog#8825). Contract only — no
behavior change, and nothing gates on freshness yet.

Adds `managedSettingsFreshness.ts`, declaring the state machine shared by
the fetch path, the policy gate and Policy Diagnostics so those consumers
cannot drift: `NotRequired` / `Pending` / `Satisfied` / `Blocked`, the
failure categories every inability-to-refresh maps to, and scoping by
account + provider + endpoint so satisfaction is never transferable
across accounts or GHE hosts.

Replaces `shouldForceRemoteSettingsRefresh` with
`resolveForceRemoteSettingsRefresh`, which resolves through
`pickManagedSettings` instead of re-implementing precedence. Two fixes
fall out: the file channel now participates (the old helper read only
native MDM and server, silently ignoring managed-file delivery), and an
explicit managed `false` is now distinguishable from an absent value,
which a later change needs in order to know when the requirement may be
cleared.

The old helper had no production caller — it was left orphaned when
661f18fdeb reworked the managed-settings fetch — so this is inert.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* policy: enforce freshness invariants in the type

Address PR feedback: `IManagedSettingsFreshness` was a bag of optional
fields, so a consumer could construct `Blocked` with no failure,
`Satisfied` with no scope, or attach `httpStatus`/`retryAfter` to states
where they mean nothing — leaving the fetch, gate and diagnostics
consumers free to drift despite the type.

Models it as a discriminated union instead, so each active state requires
the fields its contract defines. `Blocked` is itself a union keyed on the
failure category, so a status code is required for an HTTP error, a
backoff deadline for rate limiting, and neither is accepted elsewhere.
`source` is now the shared `ManagedSettingsChannel` rather than `string`,
and is required on the effective states, which also encodes that it is
never `'none'` once a channel has supplied the control.

Adds `@ts-expect-error` coverage for the three rejected shapes: the
directives fail the build if any shape becomes constructible again.

`isSameManagedSettingsFreshnessScope` is now a private helper with
required arguments — the union guarantees a scope is present, so its
undefined-tolerance was unreachable, and nothing outside this module
used it.

Also trims two over-long comments flagged in review.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chat: fail closed on forced managed settings refresh

Require a fresh managed-settings response before enabling AI features when forceRemoteSettingsRefresh is effective. Preserve recovery through sign-in and retry, expose diagnostics, and cover native, server, file, failure, scope, and sign-out behavior.

Related to microsoft/vscode-internalbacklog#8825.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chat: tighten managed settings recovery UX

Re-render the Agents window when freshness failure details change and preserve startup notification deferral.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chat: address managed settings review feedback

Scope cached server controls before precedence, avoid expired rate-limit poll loops, and align update-required recovery guidance across workbench and Agents window UI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: await explicit managed settings recovery refresh

Classic web initialization intentionally skips the default-account fetch. Exercise the explicit refresh path before asserting the no-token fail-closed state so the browser suite observes the same lifecycle it is validating.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: provide product name in policy overlay fixture

Ensure managed-settings messages render Code - OSS instead of an undefined product label in component screenshots.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: add managed settings failure modes

Let the mock policy server return HTTP errors, malformed JSON, immediate disconnects, or no response until client timeout through presets, the GUI, and the control API.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chat: tighten forced managed settings recovery

Improve forced-refresh progress and blocked-state UX, bound automatic retries after failures, preserve the ungoverned cache path, and simplify mock policy failure controls.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chat: close managed settings dialog on retry

Start the explicit managed-settings refresh without making the dialog wait for the network request to complete.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chat: separate managed settings requirement copy

Place the organization requirement and fetch failure remediation in separate paragraphs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chat: address managed settings review feedback

Preserve cached and blocked freshness state, report failed manual syncs, retain pending mock-server edits, and include attempted scope in diagnostics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chat: reduce managed settings freshness implementation

Trim redundant contract commentary and tests, simplify refresh resolution, deduplicate failure transitions, and keep no-flag tests independent from retry bypass behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-08-26 09:10:00 -07:00
joshspicerandCopilot 1faca48a3e Add valid permissions presets to mock-policy-server (#332590)
* Add valid permissions presets to mock-policy-server

Add five new managedSettings presets exercising the SDK's managed
permissions schema (deny/ask/allow rule lists and both
disableBypassPermissionsMode values), validated against
copilot-agent-runtime's managed-settings-schema.json and rule parser:

- allow-auto-only: disableBypassPermissionsMode='allow-auto-only'
- deny-dangerous-commands: deny list blocking shell/write/domain rules
- ask-before-publish: ask list requiring approval without blocking
- lockdown-allowlist: allow list intersection combined with deny
- Clarified the existing disable-bypass-permissions description

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add workspace-scoped permission preset

Add examples using the managed permission syntax where a single leading
slash scopes file rules to the workspace root. Also correct existing
examples that described workspace-scoped rules as system paths.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Clarify permission preset scopes

Document both managed bypass permission enum values and clarify that
Write(~/**) covers the user home directory, including workspaces beneath it,
not every path outside the workspace.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-08-25 12:33:40 -07:00
joshspicerandCopilot 7013704a63 Mock policy server: Add onboarding page (#331720)
* Add mock policy server onboarding

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Clarify mock policy setup method selection

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Simplify mock policy runtime terminology

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Group mock policy setup status

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Emphasize mock policy connection status

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Move mock policy setup into modal

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Align mock policy proxy diagnostic

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-08-20 13:55:34 +00:00
joshspicerandCopilot 1c264650c7 Polish mock policy schema table layout (#331512)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-08-18 21:58:13 +00:00
Kyle Cutler 45b7c7d73c Browser: CDP proxy correctness fixes (#331085)
* Browser: CDP proxy correctness fixes

* feedback
2026-08-17 22:44:31 +00:00
joshspicerandCopilot App 8fb912f511 Mock policy server: upstream passthrough, per-endpoint mocking, request log (#330711)
* Mock policy server: upstream passthrough, per-endpoint mocking, request log

The mock policy server only worked via product.overrides.json, which limits it
to Code OSS running from sources, requires a reload after every change, and
cannot exercise a stable/Insiders build or the CLI. Support a system HTTP proxy
as a second wiring path, keeping product.overrides.json as the default.

- Proxy anything not explicitly mocked to the real API (--upstream, default
  https://api.github.com) and stream it back, so a blanket proxy rule is safe:
  only endpoints deliberately switched on are faked. Rewrites Host, strips
  hop-by-hop headers and accept-encoding, forwards Authorization untouched, and
  reports upstream failures as 502.
- Add a per-endpoint mock/passthrough toggle. Only managed settings is mocked by
  default; applying a preset implicitly switches mocking on.
- Add managed-settings disk cache clearing. A cache entry under an hour old
  makes the runtime skip the network entirely, so an override is never even
  requested. Paths verified against managed_settings_cache.rs and
  path_helpers.rs, including the COPILOT_CACHE_HOME override.
- Add a rolling request log (GET/DELETE /api/log) surfaced in the GUI, so it is
  obvious whether the client actually reached the server.
- Add realistic managed-settings presets, each validated against the schema, and
  branch-point presets for the other endpoints.
- Only warn about unknown schema keys on 2xx, and re-validate on status change:
  a 404/466/500 body is an error payload, not a policy document.
- Route GUI assets from an explicit allowlist instead of probing public/ for
  anything that looks like a file, which would otherwise shadow proxied paths.

UX:
- Make save semantics consistent: everything auto-saves, with a pill showing
  whether the editor matches what is being served.
- Surface mocked vs proxied via tab dots, a checkbox, and reactive help text.
- Add a light palette; the dark-only one declared color-scheme: light dark, so
  UA form controls rendered light on a dark page.
- Make the schema disclosure a real button with aria-expanded, add focus-visible
  styles, and expose tab state to screen readers.
- Build the validation table from DOM nodes rather than innerHTML.
- Surface save and wire failures instead of failing silently, and fall back to
  the shared endpoint definitions when the control API is unreachable.
- Answer the GUI's own favicon request so it stops appearing in the log as a
  proxied 404.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Polish mock policy server workflows

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Route runtime policy diagnostics through proxy

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Minimize runtime proxy integration

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Move runtime proxy fix to separate PR

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address mock policy server review feedback

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-08-13 23:29:50 +00:00
b5e1c2c740 test: add managed settings compatibility mock responses (#330320)
* chat: negotiate managed settings client compatibility

Report the VS Code managed-settings client version, securely transport the User-Agent, parse compatibility responses, and preserve fail-closed state across refreshes.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chat: simplify managed settings client identity

Use the existing browser-safe Editor-Version convention from productService and remove the dedicated User-Agent IPC transport.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chat: report bundled Copilot runtime version

Send the runtime version from product metadata alongside the VS Code editor identity and include both values in policy diagnostics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: align peer chat sandbox expectation

Use the shared sandbox config builder introduced on main so the peer-chat assertion follows the current sandbox semantics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chat: use standard AI feature gate for compatibility

Route managed-settings compatibility failures through the existing entitlement hidden state instead of bespoke chat and Agents-window blocking.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chat: apply compatibility to policy gate context

Drive both the standard policy-gate context and entitlement hidden state when managed-settings compatibility is rejected, and cover their combined transitions.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chat: show minimum compatible client version

Include the server-provided minimum client version in the managed-settings update notification, with a fallback for malformed responses.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chat: show compatibility warning as modal

Use the built-in VS Code modal dialog for managed-settings compatibility lockout while preserving update, learn-more, close, and keyboard-dismiss actions.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: add managed settings compatibility mock responses

Add 404 and client_update_required presets plus configurable response status support to the local policy server.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: fix managed settings mock controls

Reset schema-generated examples to a successful response and document that selected presets must be applied.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: allow managed settings client identity header

Permit Editor-Version in mock endpoint CORS preflights so local testing matches the simplified client contract.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: allow Copilot runtime identity header

Permit Copilot-Runtime-Version in mock endpoint CORS preflights alongside Editor-Version.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: expose mock response status

Show and edit each mock endpoint's HTTP response status in the GUI, validate the supported range, and autosave it with the response body.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Dmitriy Vasyura <dmitriv@microsoft.com>
2026-08-13 07:28:41 -07:00
roblourensandCopilot 9187366d2e Fix Windows batch failure propagation (#330096)
* test: probe Windows batch exit propagation

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: preserve Windows batch failure exit codes

Move failure exits outside parenthesized command blocks, where cmd.exe otherwise returns zero to the caller.

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: remove Windows exit probe

The controlled CI runs captured the failure and validated the top-level failure labels, so remove the temporary diagnostics.

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: stabilize Agent Host E2E on Windows

Keep portable shell and plugin scenarios enabled while gating three documented Windows persistence and changeset gaps.

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: gate unsupported Windows Agent Host E2E

Document and skip Windows-only scratch cleanup, custom terminal metadata, and client-plugin hook gaps.

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: gate stale Windows discard summaries

Document and skip the final tracked-change discard variant whose Windows changeset summaries do not refresh.

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-08-11 15:52:35 +00:00
roblourensandCopilot 7fa0ab3432 agentHost: Fall back to GitHub token for utility calls (#329870)
* agentHost: fall back to GitHub token for utility calls

Use the existing Copilot token flow by default, but fall back to the authenticated GitHub OAuth token when token minting is forbidden. Restore commit operation coverage and remove the obsolete known issue. (Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agentHost: scope utility token fallback

Limit GitHub OAuth fallback to commit-message generation, cover both utility authentication invalidation paths, and strip account-specific headers from recorded model responses. (Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agentHost: use GitHub token for utility calls

Remove Copilot token minting from utility model requests and authenticate them directly with the accepted GitHub OAuth token. Keep provider fixtures stable with deterministic utility stubs. (Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agentHost: allow ancillary mock responses

Return deterministic content for ancillary requests when Agent Host tests intentionally register a multi-turn default scenario. This prevents direct utility calls from failing Codex provider integration teardown. (Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-08-10 20:01:33 +00:00
roblourensandCopilot a60023fc73 agentHost: skip unaffected E2E tests in PR CI (#329879)
* agentHost: skip unaffected E2E tests in PR CI

Classify changed PR files inside the existing Electron jobs and skip the bundled-provider Agent Host E2E suite when no relevant source, SDK, build, or harness files changed.

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agentHost: fail open when change detection fails

Run the full Agent Host E2E suite when pull request file enumeration fails, keeping the classifier an optimization rather than a CI dependency.

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-08-10 11:47:49 -07:00
roblourens d9ea9f8d74 agentHost: increase Copilot E2E coverage (#329602)
* agentHost: increase Copilot E2E coverage

Add deterministic Copilot provider coverage, document opt-in product issue reproductions, and remove a parallel crash-directory startup race.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agentHost: address E2E review feedback

Strengthen tool-result assertions, normalize copied plugin paths, correct live reproduction commands, and remove the invalid multi-select coverage claim.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-08-07 17:40:08 +00:00
roblourens c69cd0da38 agentHost: summarize parallel E2E failures (#329637)
Repeat failed Mocha blocks after the parallel suite summary and wait for changeset operations before invoking them in the affected conformance test.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-08-07 17:28:39 +00:00
Rob LourensandCopilot 8831478021 agentHost: run E2E suites in parallel (#329314)
* agentHost: run E2E suites in parallel

Run the deterministic conformance and provider entrypoints concurrently while preserving serial server reuse within each suite. Integrate the parallel runner into coverage and full integration runs.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agentHost: harden parallel E2E runner

Preserve Windows argument boundaries through a PowerShell wrapper and fail coverage runs when a worker does not emit protocol-surface observations.

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agentHost: stabilize terminal clear E2E test

Wait for a marker that appears only in command output, not in the shell's echoed input, before clearing terminal state.

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-08-06 07:15:51 -07:00
Rob LourensandCopilot 3c944e7580 agentHost: replace 'real sdk' SDK tests (#329244)
* agentHost: replace gated SDK tests

Run the unique SDK compatibility checks tokenlessly in normal CI and remove the obsolete real-SDK gate and helper.\n\n(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agentHost: fix SDK tests on Windows

Launch direct SDK integration tests with the same sanitized Node-mode environment used by the production Copilot agent.\n\n(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-08-05 21:11:32 +00:00
Rob LourensandCopilot 018354116a Restructure agent host E2E tests into conformance and parity tiers (#327489)
* Restructure agent host E2E tests into conformance and parity tiers

The E2E suite ran every test once per provider, so 52 provider-invariant
tests were executed three times on each of three operating systems for a
single meaningful assertion. Split the suite into two tiers:

- conformance: provider-invariant Agent Host Protocol behavior, run once
  against a single reference provider
- parity: behavior that must be verified separately for Claude, Copilot,
  and Codex

Add an IAgentHostTarget seam so the suite can launch a non-VS Code
Agent Host Protocol implementation, keeping the tests external to the
implementation under test.

Track protocol-surface coverage (commands, notifications, and action
types observed on the wire) alongside the existing line coverage, and
check the stats in so gaps are visible in review.

Freeze the protocol/ suite: it side-loads a mock agent into the
production server, so it cannot be run against an alternate
implementation. Record the migration backlog in the E2E README.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Record request-assertion and line-ending gaps in E2E known issues

Document two structural gaps found while reviewing the record/replay
design:

- The recorded model request is normalized on write but never read back,
  so replay cannot detect regressions in prompt assembly, history
  retention, or attachment marshalling. Records the projected-assertion
  approach and why tool result text must be elided from it.
- Snapshot normalization does not handle line endings, so any snapshot
  carrying literal text can fail on Windows for reasons unrelated to the
  behavior under test.

Also note the portable-command guidance (`node -e` / a seeded script) for
scenarios that genuinely need to run a command.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback on E2E coverage tooling

- Derive the conformance suite title from the provider config instead of
  passing it separately, matching defineAgentHostE2ETests and removing a
  second source of truth for the suite name.
- Warn once when protocol-surface observations cannot be written. The
  write is still non-fatal, but a silent failure previously surfaced much
  later as the coverage script reporting a missing observation file.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Gate the Claude side-chat context test on a server-tool wiring race

`side chat receives bounded source context without copied history` fails
intermittently on Claude with `Server not found: host`, raised by the
Claude CLI when a replayed turn calls a server tool before the host's
server-tool MCP server is registered for that session.

Separating the conformance and parity tiers concentrated the Claude
parity suite into consecutive model-backed turns where interleaved
host-only tests previously spaced them out, which loses this race about
half the time. Measured at ~4 failures in 8 full-suite runs, against 0 in
6 runs before the split; the test passes in isolation and with a fresh
server per test, so it is a materialization race rather than replay or
shared-server state.

Gate it behind `sideChatServerToolWiringUnstable` and record the
measurements in KNOWN_ISSUES.md, including a note not to re-record the
capture to make it pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Link the side-chat race to its tracking issue

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Drop the side-chat gate now that the underlying race is fixed

#327560 fixed the root cause: the Claude session published the host's own
in-process `host` and `client` MCP bridges into session-scoped state, so a
peer or side chat whose query had not yet reported `host` tried to toggle
it and the CLI answered `Server not found: host`.

That is the same failure this branch quarantined, so remove the
`sideChatServerToolWiringUnstable` gate and its known-issue entry rather
than carrying a stale workaround. Verified with the gate removed: 4 clean
Claude runs and 2 clean full-suite runs (150 passing, 0 failing).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-07-27 04:33:12 +00:00
Robo 1cf2f4ee31 chore: cleanup some debt comments (#326502)
* chore: remove old context menu code from code.iss

* chore: remove old app name symlink on macOS

* chore: update comment for windows fs workaround

* chore: update issue link for context menu workaround

* chore: remove stale comment from libcxx-fetcher
2026-07-20 10:03:52 +00:00
Rob Lourens 3296e952c8 Reorganize Agent Host integration tests (#326531)
* Reorganize Agent Host integration tests

Separate protocol, provider E2E, mocked-LLM, and direct SDK test families, and split shared provider scenarios into focused suites.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address Agent Host test review feedback

Keep coverage scope metadata and replay fixture documentation aligned with the reorganized suites.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Limit Agent Host coverage to provider E2E tests

Exclude mock-agent and mocked-LLM suites so the report measures only the real server and bundled provider stacks with replayed model traffic.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Clarify Agent Host E2E test boundaries

Move synthetic-LLM suites into provider integration, flatten E2E captures, and simplify the checked-in coverage summary path without changing tests or fixture contents.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-07-20 03:48:13 +00:00
Rob LourensandCopilot 693614c9f2 Add Agent Host E2E coverage and expand protocol scenarios (#326493)
* Add Agent Host E2E coverage and scenarios

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Stabilize Agent Host E2E tests across platforms

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Gate macOS-recorded Agent Host snapshots on Windows

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Use behavior snapshots for Agent Host scenarios

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-07-19 16:34:00 +00:00
Christof Marti 23e52847b7 Proxy test with isolated process on macOS (#325774) 2026-07-15 16:44:32 +02:00
Paul ff3d330991 Bump baseline version for chat perf (#325706) 2026-07-13 22:23:44 +00:00
Paul a6d303eb43 Remove loaf count from chat perf regression metrics (#325695) 2026-07-13 21:42:54 +00:00
Dileep YavanmandhaandCopilot Autofix powered by AI 1dc485dd0c Add chat terminal sandbox smoke tests (#324848)
* Add chat terminal sandbox smoke tests

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Expand chat terminal sandbox smoke coverage

* Test terminal sandbox temp directory isolation

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-07-09 01:05:43 +00:00
Paul 90b81ef265 Fix issues with chat perf pipeline (#323917) 2026-07-01 22:10:25 +00:00
RoboandCopilot 4b6f5e55bb chore: bump electron@42.5.0 (#321629)
* chore: bump electron@42.4.0

* chore: apply temp dir workaround for short paths

* chore: use 24.15.x for CI node

* chore: update nodejs build

* chore: bump electron@42.5.0

* fix: unblock playwright install on node 24.17

Node 24.16+ made Readable pause()/resume() a no-op on destroyed streams
which makes yauzl 2.x / extract-zip 2.x and older playwright extraction
hang forever.

- extensions/copilot: add "yauzl": "^3.3.1" override (was missed by #318682)
  so electron and @vscode/vsce no longer resolve the broken yauzl 2.10, fixing the
  hung `npm ci` in the Copilot and Extract chat-lib pipelines.
- extensions/copilot: bump electron ^39.8.5 -> ^42.5.0 so its install
  script uses the native @electron-internal/extract-zip instead of extract-zip.
- bump @playwright/test ^1.56.1 -> ^1.61.1 so `playwright install`
  uses the fixed extractor, unblocking the "Download Electron and
  Playwright" step in all electron test pipelines.

* chore: update build

* agentHost: fix macOS sandbox smoke sentinel parsing

On macOS CI, the AgentHost sandbox smoke test resolves the shell to
/bin/sh, which uses the sentinel-based completion path. In that path, the
parser could consume the echoed sentinel command text
(`<<<COPILOT_SENTINEL_..._EXIT_$?>>>`) before the real numeric marker
arrived, causing a false `Exit code: -1` failure even though the command
later completed successfully.

Harden the sentinel parser to ignore echoed/non-numeric sentinel text
and use the latest complete numeric marker instead. Also force the
macOS AgentHost sandbox smoke test to use /bin/sh and assert that in the
suite log so local runs exercise the same path as CI.

Adds a regression test for echoed sentinel command text.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: update screenshot baseline after playwright bump

* chore: bump distro

* chore: fix typecheck

* chore: bump distro

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-06-29 13:53:48 +00:00
Alexandru DimaandCopilot 884913d065 build: avoid per-section Electron re-download in PR test runs (#323341)
The GitHub Actions PR test workflows run integration/smoke tests out of
sources, so each test section launches scripts/code.bat, which runs
build/lib/preLaunch.ts. Unlike the Azure Pipelines product builds, the
GitHub workflows did not set VSCODE_SKIP_PRELAUNCH, so preLaunch ran on
every section and getElectron() unconditionally deleted and re-downloaded
.build/electron each time. On Windows this races with file locks held by
the just-exited Electron process and intermittently fails the whole job
with the bare 'The system cannot find the path specified.' error.

- Set VSCODE_SKIP_PRELAUNCH=1 on the unit/integration/remote test steps of
  the win32, linux and darwin PR workflows, matching Azure Pipelines (the
  workflows already prepare node_modules, out, built-in extensions and
  Electron in dedicated steps before the tests run).
- Make getElectron() version-aware: skip the destructive re-download when
  the installed Electron already matches the expected version, falling back
  to a download on any detection failure.
- Make scripts/code.bat fail fast with a clear message when preLaunch.ts
  fails instead of falling through to launch a missing executable.
- Retry rimraf on EBUSY/EPERM (Windows file-lock codes), not just ENOTEMPTY.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-06-29 08:44:22 +00:00
Paul da4387232c Add smoke tests for model configs (#323205) 2026-06-27 17:51:29 +00:00
Josh SpicerandCopilot a4ce08f735 Refactor Copilot managed-settings for maintainability (#322439)
* Refactor Copilot managed-settings for maintainability

Centralize structured (object/array) managed-setting handling behind a
single descriptor table so adding a key touches one place, consolidate the
duplicated equality helpers onto `equals`, and add shared
`hasManagedSettingsDefinitions` and `managedSettingValue` helpers. Strictly
behavior-preserving.

Incorporates a 3-model maintainability review:

- `adaptManagedSettings` builds the scalar remainder via `{ ...response }`
  plus delete (CopyDataProperties) instead of for..in + assignment, so a
  server-sent own `__proto__` key cannot trigger the inherited setter. This
  matches the original `...rest` semantics; adds a regression test.
- `managedSettingValue` is memoized per key so its policy-definition
  reference identity is real rather than incidental to the call site.
- Corrected JSDoc and skill docs that overstated `responseField` as
  compiler-checked; it is a hand-maintained union backstopped by tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Clarify why structured managed-settings keys must declare type: 'string'

The bag-carrying `type` is load-bearing, not cosmetic: `projectManagedSettings`
gates each value with `typeof value === type` and drops mismatches, and the
native MDM watcher reads the registry/plist value as that type. Spell out that
omitting it (or declaring the object/array type) makes a structured key fail
projection and silently never apply.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review: allocation-free empty check, __proto__ test, doc accuracy

- hasManagedSettingsDefinitions: reuse the allocation-free isEmptyObject
  helper instead of Object.keys(...).length (the bot's only valid nit).
- Add a primitive `__proto__` regression test proving a server-sent
  `{"__proto__": true}` scalar is dropped, never pollutes the result
  (disproves the reviewer's prototype-pollution concern).
- Fix github-managed-settings.md: omitting `type` or declaring
  `'object'`/`'array'` is a compile error (the field is required and
  constrained to `'string' | 'number' | 'boolean'`), not a runtime drop;
  only `'number'`/`'boolean'` compile-but-drop-at-runtime.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Surface managed-settings source in Policy Diagnostics

Centralize the server-over-MDM precedence into a shared selectManagedSettings
helper (plus a ManagedSettingsSource union) and reuse it in both
AccountPolicyService and the Policy Diagnostics report, so the report can never
drift from the source that policy evaluation actually applies.

Rewrite the diagnostics "Managed Settings" section to:
- show the Active source (GitHub Server API / Native MDM / None)
- break down each channel (server fetch status + raw response, native MDM bag)
- label the raw response as the last *successful* fetch, so a later failed
  fetch (e.g. a 404) no longer looks like it contradicts an empty effective bag
- compute the true effective bag via the shared projectManagedSettings

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix mock policy server "Generate example" not persisting

The "Generate example" button filled the editor and the localStorage draft but
never called debouncedSave(), so the generated body was never POSTed to
/api/state and the endpoint kept serving the empty preset. Add the missing
debouncedSave() to match applyPreset() and the editor input handler.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Strip prose from Policy Diagnostics and collect managed-settings parse errors

The Developer: Policy Diagnostics "Managed Settings" section now renders
data only (tables and JSON blocks, no explanatory paragraphs).

It also collects non-fatal parsing/normalization warnings from every stage
of the managed-settings pipeline, jsonc-style (accumulate, never throw), and
surfaces them in a new "Parse Errors" section:
- adapt: re-runs adaptManagedSettings on the raw server response
- project: re-runs projectManagedSettings against the declared policy keys
- parse: re-parses JSON-payload string values with the jsonc parser

This explains why a key is silently dropped. For example a server
extraKnownMarketplaces entry with source "github" but no "repo" now shows
the "requires \"repo\"" warning instead of just vanishing from the bag.

Adds a focused test for that github-without-repo normalization case.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Tighten Policy Diagnostics managed-settings rendering (review follow-up)

Code-quality pass on the managed-settings diagnostics section:

- Extract a jsonBlock() helper for the repeated fenced-JSON rendering
  (4 call sites collapsed).
- Parse only the known JSON-payload keys (enabledPlugins,
  strictKnownMarketplaces, extraKnownMarketplaces) instead of a
  leading-brace heuristic. This mirrors what PolicyConfiguration actually
  parses, avoids mis-sniffing scalar values, and catches malformed payloads
  that don't start with a brace.
- Unify the raw-response guard on isObject() so the printed raw response and
  the adapt-stage warning harvest use one predicate.
- Drop the defensive object copy in projectManagedSettings(); it is read-only,
  so normalize undefined with `?? {}` instead of spreading.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix native MDM availability in Policy Diagnostics; tidy table headers

The diagnostics report showed "Native MDM | Available | no (desktop only)"
even on desktop. ICopilotManagedSettingsService was registered only in the
electron-main process and hand-plumbed into AccountPolicyService, but never
placed in the renderer service collection, so the report's
accessor.get(ICopilotManagedSettingsService) always threw and mislabeled the
channel as unavailable.

Register the CopilotManagedSettingsChannelClient (the renderer's handle to the
main-process service) in the service collection in both desktop.main.ts and
sessions.main.ts. The diagnostics now resolves it on desktop and Agents windows
and reports real native MDM availability and values; web still has no native
channel and correctly reports unavailable.

Also tidy the report builder: extract a PROPERTY_VALUE_TABLE_HEADER constant for
the five repeated two-column table headers, and drop the now-misleading
"(desktop only)" annotation on the availability row.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-06-22 17:42:10 -07:00
Josh SpicerandCopilot 593c7f2366 policy: dev mock server for copilot_internal policy endpoints (#321871)
* policy: add dev mock server for copilot_internal policy endpoints

Adds scripts/mock-policy-server, a standalone dev tool (npm run
mock-policy-server) that mocks the Copilot policy endpoints
DefaultAccountService calls: entitlements (/copilot_internal/user), token
(/copilot_internal/v2/token), MCP registry (/copilot/mcp_registry) and
managed settings (/copilot_internal/managed_settings).

A small web GUI lets devs pick presets or edit each JSON response, and
Wire/Unwire buttons point product.overrides.json at the local server
(preserving the rest of defaultChatAgent, since bootstrap-meta merges
overrides shallowly). The managed-settings JSON schema is loaded from
--schema/MANAGED_SETTINGS_SCHEMA, defaulting to
./copilot-agent-runtime/schema/managed-settings-schema.json relative to
the app cwd; web URLs and file URIs are accepted, and the GUI warns about
keys not declared in the schema.

The three browser/shared .js files are added to
.eslint-allowed-javascript-files since the GUI loads them directly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* policy: address mock-policy-server review feedback

- Scope permissive CORS to the mocked GET endpoints only; keep /api/*
  same-origin so a website can't drive /api/wire and rewrite
  product.overrides.json (CSRF).
- Coerce an empty editor body to {} instead of "" so mocked responses
  stay JSON objects.
- Build the endpoint meta line with textContent/DOM nodes instead of
  innerHTML.
- Drop the misused tablist/tab ARIA roles; the nav now has an aria-label
  and the active item uses aria-current.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* policy: document mock policy server in add-policy skill

Add local-testing.md to the add-policy skill with basic steps for using
the mock policy server (scripts/mock-policy-server) to exercise the
account/managed-settings flow locally, and link it from SKILL.md and
github-managed-settings.md.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* policy: polish mock server GUI — schema validation, wiring backup, localStorage persistence

* policy: auto-save, rename wiring to product.overrides.json, copy path button

* mock-policy-server: convert server.js to TypeScript; add raw response diagnostics

- Convert server.js → server.ts (runs via --experimental-strip-types)
- Add endpoints.d.ts type declarations for the UMD endpoints module
- Add managedSettingsRawResponse to IDefaultAccountProvider/IDefaultAccountService
- Show raw response in Developer: Sync Account Policy output
- Remove server.js from eslint allowed-javascript-files

* mock-policy-server: convert all JS to TypeScript

- endpoints.js → endpoints.ts with proper interfaces (replaces .d.ts)
- public/app.js → public/app.ts with full type annotations
- Server uses module.stripTypeScriptTypes() to serve .ts as plain JS
  to the browser — no build step needed
- Remove all mock-policy-server entries from .eslint-allowed-javascript-files

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-06-18 21:55:01 +00:00
Christof Marti 6195d33f63 SDK sandbox and tests 2026-06-11 16:39:30 +02:00
BeniBenj d2c49fa66a update AHP version 2026-06-09 23:48:23 +02:00
Alex Ross 7c1dd895f5 Convert mock-llm-server.js to ts (#320567)
* Convert mock-llm-server.js to ts

* Ad eslint rule for no new js files

* Message
2026-06-09 15:08:47 +02:00
Alexandru Dima ff61118bc0 fix: support token auth for CLI SDK mock server to enable auto-model in smoke tests (#320072)
* fix: support token auth for CLI SDK mock server to enable auto-model in smoke tests

- Add `advanced.debug.overrideAuthType` setting to control HMAC vs token
  auth when overrideProxyUrl is set (default: HMAC for dev, token for tests)
- Update mock server model definitions to match real CAPI response shape
  (family, vendor, version, supported_endpoints, billing, etc.)
- Add `selected_model` to mock `/models/session` response (required by SDK
  auto-mode resolution)
- Add Responses API SSE handler for gpt-5.3-codex which uses `/responses`
  instead of `/chat/completions`
- DRY up mock model definitions with shared `ALL_MODELS` array

* fix: add inspectConfig to test mock for copilotCliAuth and skip the other CLI smoke tests for now

* Don't run in PRs for now
2026-06-05 13:52:11 +02:00
Christof Marti 837d9b877e Add Copilot CLI sandbox test (#317981) 2026-06-04 15:12:55 -07:00
Alex Ross 616eab07e3 Add "hello" tests to VS Code smoke tests (#319719)
* Add "hello" tests to VS Code smoke tests

* Address CCR feedback
2026-06-03 15:29:51 +02:00
Paul 4810972048 Run chat performance pipeline on commit (#318868) 2026-05-28 20:35:17 -07:00
Anthony Kim 3827a8b38a Smoke test for Copilot CLI via chat extension in editor window (#317140)
* Copilot CLI sanity testing

* Make things better

* see if 1.0.48 correctly fails

* Try to be smarter with git auth

* Test  if copilot cli sanity test auth correctly.

* Test if copilot cli sanity test FAILS correctly

* Try to get sanity tests pass?

* Copilot CLI from chat extension sanity test

* Try more smoke test for copilot cli

* More smoke test related things?

* title is messing with chat disabled..?

* Why is there timeout for .editor-instance .interactive-session

* add copilot cli ui smoke auth diagnostics

* Be more descriptive when copilot cli smoke test fail

* remove integration test, upgrade smoke test

* Clean up Copilot CLI smoke test diagnostics

* Get even more inspiration from agent smoke test
2026-05-21 16:00:35 -07:00
Sandeep SomavarapuandCopilot 3597bff21d fix and enable agents window smoke tests (#317764)
* add logs

* fix _currentNewSession race in async send flows

Async commit-wait flows (_sendFirstChat, _sendFirstChatViaController,
_sendSubsequentChat) unconditionally cleared _currentNewSession on
completion. When a newer session was created while the previous one was
still awaiting commit, the clear stomped the newer session's pointer —
causing 'Session not found' errors on the next send.

Extract _clearCurrentNewSessionIfMatch() that only clears when the
value still points at the session that initiated the async flow.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* enable claude test

* update logging

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-21 15:48:29 +02:00
2e8b995ab1 agents: add smoke test for Agents Window with mocked LLM server (#317545)
* agents: add smoke test for Agents Window with mocked LLM server

Adds a smoke test that opens the Agents Window, creates a new session
on a workspace folder, sends a 'hello world' prompt, and verifies the
request reaches a local mock LLM server that returns a canned response.

The test exercises two session types: Copilot CLI and Claude Code. A
third Local-session test is included but marked `it.skip` for now.

Key pieces:

* `test/smoke/src/areas/agents/agents.test.ts` — new smoke suite. Starts
  `scripts/chat-simulation/common/mock-llm-server.js` on a random port,
  registers a per-test scenario with a distinct reply, and injects
  `VSCODE_COPILOT_CHAT_TOKEN` / `GITHUB_PAT` / `IS_SCENARIO_AUTOMATION`
  env vars so the Copilot extension's token manager picks up a fake
  token whose endpoints.api/proxy point at the mock server.

* `test/automation/src/agents.ts` — new `Agents` workbench helper with
  `openCurrentFolderInAgentsWindow`, `switchToAgentsWindow`,
  `startNewSession`, `selectSessionType`, `submitNewSessionPrompt`,
  `waitForAssistantText`.

* `test/automation/src/code.ts` + `electron.ts` — `LaunchOptions` now
  accepts an `extraEnv` map that is merged on top of `process.env`
  when spawning the Electron child, so tests can inject env-based mocks
  without going through a custom launcher.

* `src/vs/sessions/browser/sessionsSetUpService.ts` —
  `shouldSkipSessionsWelcome` now returns `true` whenever
  `enableSmokeTestDriver` is set, so the welcome/auth dialog does not
  block smoke runs.

* `scripts/chat-simulation/common/mock-llm-server.js` — adds two
  models to `EXTRA_MODELS` (`gpt-5.3-codex` for Copilot CLI default,
  `claude-sonnet-4.5` for Claude Code), and routes `/v1/messages` to a
  new `handleMessagesApi` that streams Anthropic-format SSE
  (`message_start` / `content_block_delta` / `message_stop`) which the
  Claude Code session type's messages-API parser expects.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fix copilot cli test

* skip claude test

* sessions: fix new-session removal regression and rename smoke test

1. Revert the `_refreshSessionCache` filter change from d5747b31c4
   back to `adapter instanceof AgentSessionAdapter`. The broadened
   `adapter !== this._currentNewSession` check raced with the
   unconditional `this._currentNewSession = undefined` in the
   `_sendFirstChat*` paths: a late callback from a previous session's
   commit would wipe the pointer and the next refresh would evict the
   new session's temp adapter, navigating the Agents Window back to the
   homepage mid-request.

2. Rename the Agents Window smoke test infrastructure for clarity:
   - `test/automation/src/agents.ts` -> `agentsWindow.ts`
     (class `Agents` -> `AgentsWindow`,
     `workbench.agents` -> `workbench.agentsWindow`)
   - `test/smoke/src/areas/agents/agents.test.ts` ->
     `areas/agentsWindow/agentsWindow.test.ts`

Verified with 10 consecutive smoke-test loops: 8/10 fully green
(Copilot CLI + Claude + Local), 2 intermittent UI timing flakes
unrelated to the navigation regression.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agents smoke: address PR #317545 review feedback

- switchToAgentsWindow: replace fixed 2s sleep with waitForElement on
  the Agents workbench DOM (`.agent-sessions-workbench`) so the helper
  returns as soon as the new window is interactable.
- resolveElectronConfiguration: apply `extraEnv` last, after the
  TESTRESOLVER_* assignments in the remote branch, so caller-provided
  env vars truly have final precedence.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agents smoke: skip Agents Window tests for OSS quality

The Copilot extension is not built in OSS CI (it's in excludedExtensions
and its dist/extension.js is only produced by its own esbuild pipeline).
Without it all three session-type providers fail to activate, causing
every Agents Window test to time out.

Skip the suite when quality is OSS, matching the pattern used by
setupExtensionTests and setupLocalizationTests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* copilotcli: add proxy endpoint to getAuthInfo for mock server routing

Without `endpoints.proxy`, the SDK's model-fetch calls
(`/models/session`, `/copilot_internal/v2/token`) fall back to
the real GitHub API which rejects the fake HMAC with a 401. This
caused intermittent smoke test failures (1 in 10) because the
Copilot CLI language models never registered, making the chat-setup
readiness gate depend on Claude's model registration timing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* copilotcli: don't cache failed model fetches

When `getAvailableModels` throws (e.g. transient network failure or
HMAC validation error with a proxy), the empty result was permanently
cached in `_availableModels`. Subsequent calls to `getModels()` would
return the cached empty array without retrying, leaving the Copilot CLI
language model provider with zero models for the rest of the session.

Clear `_availableModels` on error so the next call retries the fetch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* agents smoke: reset workspace before opening Agents Window

Earlier smoke test suites (e.g. Tasks) modify .vscode/tasks.json and
leave uncommitted changes. A dirty workspace prevents worktree creation
and triggers the uncommitted-changes confirmation flow which aborts the
Copilot CLI session on builds.

Reset via `git checkout . --quiet` in the before hook, matching the
pattern used by notebook and search smoke tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Sandeep Somavarapu <sandy081@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-05-20 13:34:54 -07:00
Connor Peet 9852223be3 agentHost: apply organized AHP types
Brings in https://github.com/microsoft/agent-host-protocol/pull/132. No runtime changes.
2026-05-19 11:12:17 -07:00
Paul cc33dc6e7c Fixes for the chat perf pipeline (#311605) 2026-04-21 19:11:30 -07:00
Paul ec992baa49 Add performance tests (#309700) 2026-04-17 21:23:43 +00:00
Alex Rossandgithub-actions[bot] 027a4d3ce4 Bump version to 1.117.0 (#309394)
* Bump version to 1.117.0

* npm i

* wait to do engine version bump

* Revert "wait to do engine version bump"

This reverts commit 9db1c0feb6.

* Add Copilot extension tests to Linux/Windows Electron integration test runs

* Remove failing step that we moved to the main build

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-04-13 14:52:32 +02:00
Peng Lyu a4855ab045 agentHost: support --host and print resolved server urls (#306219) 2026-03-30 14:23:52 -07:00
Rob Lourens c065b175fd Add argument parsing, suite filtering, and grep support to integration test scripts (#305837)
* Add argument parsing, suite filtering, and grep support to integration test scripts

- Add --run, --runGlob, --grep, --suite, and --help argument parsing
- --suite selects extension host test suites (comma-separated, glob patterns)
- --grep forwards test name filter to all runners via MOCHA_GREP env var
- Validate --suite filter matches at least one known suite
- Add MOCHA_GREP support to testrunner.js, CSS and HTML test runners
- Seed user settings to suppress dock bounce notifications
- Always apply *.integrationTest.js glob for node.js tests
- Add integration-tests skill documentation

* Address Copilot review feedback

- Quote cd $ROOT, rm -rf $VSCODEUSERDATADIR, rmdir %VSCODEUSERDATADIR%
- Quote --runGlob pattern to prevent premature glob expansion
- Use GREP_ARGS array for safe grep forwarding in .sh
- Use conditional call with proper quoting for grep in .bat
- Deduplicate suite list into KNOWN_SUITES variable
- Remove unused EXTRA_ARGS and ARGS variables from .bat

* Fix Windows CI: remove unnecessary enabledelayedexpansion

The original script used plain 'setlocal'. Adding 'enabledelayedexpansion'
may affect path resolution behavior on Windows CI. Since no delayed
expansion (\!var\!) syntax is used, revert to the original 'setlocal'.

* Fix Windows CI: capture %~dp0 before call :label corrupts it

In Windows batch, 'call :label' can change what %~dp0 resolves to.
Our should_run_suite subroutine uses 'call :should_run_suite', which
caused %~dp0 to resolve to the wrong directory for extension paths
that appear after the subroutine call. Capture the script directory
once at startup into %SCRIPT_DIR% and use it everywhere.
2026-03-28 11:23:37 +11:00