Commit Graph

122 Commits

Author SHA1 Message Date
Rob Lourens
c065b175fd Add argument parsing, suite filtering, and grep support to integration test scripts (#305837)
* Add argument parsing, suite filtering, and grep support to integration test scripts

- Add --run, --runGlob, --grep, --suite, and --help argument parsing
- --suite selects extension host test suites (comma-separated, glob patterns)
- --grep forwards test name filter to all runners via MOCHA_GREP env var
- Validate --suite filter matches at least one known suite
- Add MOCHA_GREP support to testrunner.js, CSS and HTML test runners
- Seed user settings to suppress dock bounce notifications
- Always apply *.integrationTest.js glob for node.js tests
- Add integration-tests skill documentation

* Address Copilot review feedback

- Quote cd $ROOT, rm -rf $VSCODEUSERDATADIR, rmdir %VSCODEUSERDATADIR%
- Quote --runGlob pattern to prevent premature glob expansion
- Use GREP_ARGS array for safe grep forwarding in .sh
- Use conditional call with proper quoting for grep in .bat
- Deduplicate suite list into KNOWN_SUITES variable
- Remove unused EXTRA_ARGS and ARGS variables from .bat

* Fix Windows CI: remove unnecessary enabledelayedexpansion

The original script used plain 'setlocal'. Adding 'enabledelayedexpansion'
may affect path resolution behavior on Windows CI. Since no delayed
expansion (\!var\!) syntax is used, revert to the original 'setlocal'.

* Fix Windows CI: capture %~dp0 before call :label corrupts it

In Windows batch, 'call :label' can change what %~dp0 resolves to.
Our should_run_suite subroutine uses 'call :should_run_suite', which
caused %~dp0 to resolve to the wrong directory for extension paths
that appear after the subroutine call. Capture the script directory
once at startup into %SCRIPT_DIR% and use it everywhere.
2026-03-28 11:23:37 +11:00
Ben Villalobos
b5903ae54c Support folding in git COMMIT_MSG files (#272356) 2025-10-21 09:26:47 -07:00
Benjamin Pasero
367fc5f243 exp - introduce and use flag to disable experiments (#254057)
* exp - introduce and use flag to disable experiments

* fix
2025-07-04 09:17:56 +00:00
Benjamin Pasero
2368e6dc36 Revert "tests - enable API tests with test runner" (#242688)
Revert "tests - enable API tests with test runner (#241149)"

This reverts commit 8f2e1b22ed.
2025-03-05 15:39:39 +01:00
Benjamin Pasero
8f2e1b22ed tests - enable API tests with test runner (#241149)
* Revert "Revert "tests - enable API tests with test runner" (#241146)"

This reverts commit 105d4af21c.

* update

* try to fix

* fix: set extension development path for API tests

---------

Co-authored-by: Connor Peet <connor@peet.io>
2025-03-03 15:43:29 -08:00
Benjamin Pasero
105d4af21c Revert "tests - enable API tests with test runner" (#241146)
Revert "tests - enable API tests with test runner (#240876)"

This reverts commit 0c4d85850f.
2025-02-19 07:15:40 +01:00
Benjamin Pasero
0c4d85850f tests - enable API tests with test runner (#240876)
* tests - enable API tests with test runner

* windows

* use correct specific tests per api workspace type

---------

Co-authored-by: Connor Peet <connor@peet.io>
2025-02-18 22:15:32 +01:00
Daniel Imms
9e688d92ba Add terminal suggest tests to run on Windows
Part of #234777
2024-11-27 11:32:49 -08:00
Martin Aeschlimann
f8e51fe82d integration tests: colorization test are run 6 times (#230380)
* integration tests: colorization test are run 6 times

* adding missing --
2024-10-03 12:34:14 +02:00
Benjamin Pasero
c5b6ef14e6 esm - more cleanup (#230285) 2024-10-02 11:55:17 +02:00
Robo
b5a6aa14a8 feat: switch to npm as default package manager (#226927)
* feat: move from yarn to npm

* chore: skip yarn.lock files

* fix: playwright download

* chore: fix compile and hygiene

* chore: bump vsce@2.17.0

Refs 8b49e9dfdf

* test: update results for bat and sh colorizer tests

* fix: add missing lock files for windows

* fix: switch to legacy-peer-deps

* chore: update markdown-it@14.1.0

Refs 737c95a129

esbuild step in extensions-ci-pr was previously using markdown-it
from root which had userland punycode and was able to compile successfully.

* ci: increase pr timeout for windows integration tests

* chore: fix product build

* build: ignore extension dev dependency for rcedit

* build: fix working directory inside container

* build: fix dependency generation

* npm: update dependencies

* ci: use global npmrc

* ci: update cache

* ci: setup global npmrc for private npm auth

* build: fix extension bundling

* chore: sync npm dependencies

* ci: debug env variables for container

* ci: fix win32 cli pipeline

* build: fix npmrc config usage for build/ and remote/ dirs

* fix: windows build

* fix: container builds

* fix: markdown-language-features tests and bundling

```
[03:58:22] Error: Command failed: /Users/demohan/.nvm/versions/node/v20.15.1/bin/node /Users/demohan/github/vscode/extensions/markdown-language-features/esbuild-notebook.js --outputRoot /Users/demohan/github/vscode/.build/extensions/markdown-language-features
✘ [ERROR] Could not resolve "punycode"

    extensions/markdown-language-features/node_modules/markdown-it/lib/index.js:14:27:
      14 │ var punycode     = require('punycode');
         ╵                            ~~~~~~~~~~

  The package "punycode" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
```

Adds userland package based on beed9aee2c

* fix: container builds for distro

* chore: update yarn occurrences

* fixup! chore: bump vsce@2.17.0

Uses the closest version to `main` branch that does not
include d3cc84cdec
while still having the fix 8b49e9dfdf

* chore: sync npm dependencies

* chore: sync npm dependencies

* chore: sync npm dependencies

* chore: throw error when yarn is used for installation

* chore: add review feedback

* chore: switch exec => run where needed

* chore: npm sync dependencies

* fix: markdown-language-features bundling

```
✘ [ERROR] Could not resolve "punycode"

    extensions/markdown-language-features/node_modules/markdown-it/lib/index.js:14:27:
      14 │ var punycode     = require('punycode');
         ╵                            ~~~~~~~~~~

  The package "punycode" wasn't found on the file system but is built into node. Are you trying to bundle for node? You can use "platform: 'node'" to do that, which will remove this error.
```

Adds missing userland package based on markdown-it/markdown-it@beed9ae,
can be removed once we update markdown-it >= 14.1.0

* ci: rename no-yarn-lock-changes.yml

* chore: sync npm dependencies

* ci: restore no-yarn-lock-changes.yml

We can disable it in a separate PR to keep the required
checks happy and also need workflow edit perms.

* chore: sync npm dependencies

* ci: rebuild cache

* ci: fix no-package-lock-changes.yml

* chore: bump distro

* chore: rm yarn.lock files

* chore: rm yarn.lock files without dependencies

* chore: add vscode-selfhost-import-aid to postinstall dirs

* chore: bump distro
2024-09-06 22:18:02 +09:00
Martin Aeschlimann
31e5940fb0 adopt test-extension for colorize & configuration-editing (#195691)
* adopt test-extension for  colorize & configuration-editing

* update

* fixes
2023-11-17 20:22:36 +01:00
Tyler James Leonhardt
482d5ba393 Add some integration tests for github-auth (#195729) 2023-10-16 13:58:44 -07:00
Aaron Munger
583d56685f use test extension to test extensions 2023-10-13 15:55:28 -07:00
Connor Peet
c2a6932e9e eng: onboard to the extension test runner (#195570)
* eng: onboard to the extension test runner

Adds a `.vscode-test.js` file that uses the new extension test CLI to
run tests. Also, onboards the markdown-language-features as the first
built-in extension to use it.

With the `ms-vscode.extension-test-runner` extension installed, the
markdown-language-features' tests can be run and debugged easily in
the UI :)

* fixup
2023-10-13 13:11:05 -07:00
Tyler James Leonhardt
a4a2d8f0f4 Rename --disable-keytar to --use-inmemory-secretstorage (#192312)
This renames the flag as the title suggests but keeps `--disable-keytar` in the `deprecates` section so it still works.

Fixes https://github.com/microsoft/vscode/issues/191498
2023-09-08 17:58:32 +02:00
aamunger
93b914bec0 run notebook renderer tests 2023-06-05 10:43:54 -07:00
aamunger
0117c714c0 remove broken notebook renderer tests from script 2023-03-17 13:16:09 -07:00
aamunger
11c5e141ab added notebook output renderer tests 2023-03-16 12:10:24 -07:00
Benjamin Pasero
03ca1e0d40 tests - align tests on all platforms (#161696)
* web - run ipynb tests on windows too

* some cleanup

* run all tests

* more cleanup
2022-09-26 09:16:42 +02:00
Rob Lourens
c717237d7c Reenable a couple notebook smoketests (#159521)
* Remove vscode-notebook-tests in favor of an .ipynb in vscode-smoketest-express

* Update build folder

* Add build task to correct platform

* Build for smoke tests on other platforms

* Fix repo url and remove comment

* Just -media?

* Update darwin/win32 as well
2022-09-14 13:22:16 -07:00
Matt Bierner
bbe7b54cf4 Remove custom editor tests (#152777)
These tests are not actively being run or maintained. Removing them for now
2022-06-21 15:37:19 -04:00
Martin Aeschlimann
c46555d9c8 Tests for configuration editing completion providers (#151953)
tests for configuration editing completion providers
2022-06-14 09:08:27 +02:00
Ladislau Szomoru
d1f882f635 Engineering - Exploration to run tests in parallel (#149698) 2022-05-18 10:37:44 +02:00
Sandeep Somavarapu
05b9eddd4b - override extension type of extensions under dev from existing extensions
- validate after updating the type
2022-05-02 14:05:30 +02:00
Benjamin Pasero
c64876aff9 💄 test scripts (#141791)
* try it

* log window ready states as info

* 💄

* compile more

* better format

* fix

* are you serious?

* bumpb

* .

* .
2022-01-31 07:28:38 +01:00
Benjamin Pasero
bef4dba21f Parcel update and CI improvements (#136830)
* 🆙 product

* 🆙 parcel/watcher

* 🆙 parcel/watcher

* tests - separate remote logs from others

* store server logs too

* more tweaks

* fix name

* error when 10s passed
2021-11-10 08:13:56 +01:00
Sandeep Somavarapu
cae4478f04 enable proposed api flag 2021-10-08 09:33:16 +02:00
Benjamin Pasero
3a4e16cac1 tests - skip release notes 2021-09-03 11:32:11 +02:00
Rob Lourens
ffb40deb52 Build ipynb extension before integration test 2021-08-03 12:15:16 -07:00
Benjamin Pasero
5aa1570976 tests - align integration test scripts 2021-07-01 15:23:37 +02:00
Rob Lourens
526253bc34 Remove unneeded vscode-notebook-tests references
Fix #126384
2021-06-17 20:51:13 -07:00
Joao Moreno
b79d23d356 :table_flip: 2021-06-15 17:07:53 +02:00
Benjamin Pasero
3f4039d99f . 2021-06-15 15:45:10 +02:00
Benjamin Pasero
270002e3b8 fix emmet test integration path 2021-06-15 15:40:24 +02:00
Benjamin Pasero
2ddc1e4fc5 Produce logs during test runs (integration, smoke) and publish as artefacts (#126065)
* first cut upload log files from integration test runs

* always publish log files

* move logs into integration-tests folder

* also produce logs for smoke tests

* store remote log files

* wire in logspath arg properly

* always publish logs

* enable logs for browser based smoke tests

* log only in verbose
2021-06-15 08:33:26 +02:00
Benjamin Pasero
33f84b36fd tests - disable getting started in integration tests and smoke tests (#125808)
* tests - disable getting started in integration tests and smoke tests

* --skip-getting-started => --skip-welcome

Co-authored-by: Jackson Kearl <jakearl@microsoft.com>
2021-06-09 18:36:42 +00:00
Benjamin Pasero
71490cd84c windows integration tests - use --no-sandbox (#125733) 2021-06-08 19:12:33 +02:00
Ladislau Szomoru
b0509196f6 Disable workspace trust during tests 2021-06-03 10:21:19 +02:00
Matt Bierner
a87db274da Make sure we build extension media for integration tests
For #120213
2021-04-23 16:18:15 -07:00
Matt Bierner
4feadc3998 Fixes #112164 2021-02-19 17:22:38 -08:00
Benjamin Pasero
9de38fc9a8 Disable keytar in our integration tests (#116852)
* tests - extract and use ALL_PLATFORMS_API_TESTS_EXTRA_ARGS

* tests - throw when accessing keytar from API tests
2021-02-17 15:09:40 +01:00
João Moreno
0e7f3d0d8c Revert "remove colorize-tests extension usage"
This reverts commit d577c4b18d.
2021-01-19 11:39:39 +01:00
Matt Bierner
800e173c40 Split ts into unit and smoke tests
The smoke tests are less reliable as they simulate user actions

Fixes #102075
Fixes #102097
Fixes #102106
2021-01-15 13:32:27 -08:00
João Moreno
d577c4b18d remove colorize-tests extension usage 2021-01-15 12:59:42 +01:00
Martin Aeschlimann
3e8b2d86d1 bring back css integration tests 2020-12-18 17:38:23 +01:00
Martin Aeschlimann
8dc8025cd4 comment out css tests on windows 2020-12-18 14:30:23 +01:00
Matt Bierner
6ad6905f98 Re-enable *.integrationTest on windows
Fixes #112164
2020-12-09 14:08:15 -08:00
Ladislau Szomoru
bd364ea311 Run commosJS integration tests consistently (#104914) 2020-08-18 12:04:38 +02:00
Matt Bierner
bdd3721849 Add custom editor test extension
Adds a simple set of tests for custom editors in a new extension. This is currently not run during CI since we want more testing to make sure it is reliable
2020-07-29 14:05:44 -07:00