Microsoft/vscode
1
0
Fork 0
mirror of https://github.com/microsoft/vscode.git synced 2026-05-08 09:08:48 +01:00
Code Issues Packages Projects Releases Wiki Activity

Web: run integration tests as part of our product builds (fix #83923)

Browse Source
This commit is contained in:
Benjamin Pasero
2020-02-10 11:22:55 +01:00
parent 16954b49fd
commit 9d9d729655
18 changed files with 161 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/README.md
+3 -30
View File
@@ -3,34 +3,7 @@
## Contents
This folder contains the various test runners for VSCode. Please refer to the documentation within for how to run them:
* `unit`: our suite of unit tests
* `integration`: our suite of API tests
* `smoke`: our suite of automated UI tests
* `unit`: our suite of unit tests ([README](unit/README.md))
* `integration`: our suite of API tests ([README](integration/browser/README.md))
* `smoke`: our suite of automated UI tests ([README](smoke/README.md))
* `ui`: our suite of manual UI tests
### Unit Tests (Electron-runner)
```
./scripts/test.[sh|bat]
```
All unit tests are run inside a electron-browser environment which access to DOM and Nodejs api. This is the closest to the enviroment in which VS Code itself ships. Notes:
- use the `--debug` to see an electron window with dev tools which allows for debugging
- to run only a subset of tests use the `--run` or `--glob` options
### Unit Tests (Browser-runner)
```
yarn test-browser --browser webkit --browser chromium
```
Unit tests from layers `common` and `browser` are run inside `chromium`, `webkit`, and (soon’ish) `firefox` (using playwright). This complements our electron-based unit test runner and adds more coverage of supported platforms. Notes:
- these tests are part of the continuous build, that means you might have test failures that only happen with webkit on _windows_ or _chromium_ on linux
- you can these tests locally via yarn `test-browser --browser chromium --browser webkit`
- to debug, open `<vscode>/test/unit/browser/renderer.html` inside a browser and use the `?m=<amd_module>`-query to specify what AMD module to load, e.g `file:///Users/jrieken/Code/vscode/test/unit/browser/renderer.html?m=vs/base/test/common/strings.test` runs all tests from `strings.test.ts`
- to run only a subset of tests use the `--run` or `--glob` options
test/integration/browser/README.md
+14 -8
View File
@@ -1,13 +1,19 @@
# VS Code Integration test
# Integration test
### Run
## Compile
```bash
Make sure to run the following command to compile and install dependencies:
# Dev (Electron)
scripts/test-integration.sh
yarn --cwd test/integration/browser
# Dev (Web)
node test/integration/browser/out/index.js
## Run (inside Electron)
```
scripts/test-integration.[sh|bat]
All integration tests run in an Electron instance. You can specify to run the tests against a real build by setting the environment variables `INTEGRATION_TEST_ELECTRON_PATH` and `VSCODE_REMOTE_SERVER_PATH` (if you want to include remote tests).
## Run (inside browser)
resources/server/test/test-web-integration.[sh|bat] --browser [chromium|webkit]
All integration tests run in a browser instance as specified by the command line arguments.
test/integration/browser/package.json
+3 -1
View File
@@ -13,6 +13,8 @@
"@types/tmp": "^0.1.0",
"rimraf": "^2.6.1",
"tmp": "0.0.33",
"typescript": "3.7.5"
"tree-kill": "1.2.2",
"typescript": "3.7.5",
"vscode-uri": "2.1.1"
}
}
test/integration/browser/src/index.ts
+40 -24
View File
@@ -9,11 +9,13 @@ import * as playwright from 'playwright';
import * as url from 'url';
import * as tmp from 'tmp';
import * as rimraf from 'rimraf';
import { URI } from 'vscode-uri';
import * as kill from 'tree-kill';
const optimist = require('optimist')
.describe('workspacePath', 'path to the workspace to open in the test').string()
.describe('extensionDevelopmentPath', 'path to the extension to test').string()
.describe('extensionTestsPath', 'path to the extension tests').string()
.describe('workspacePath', 'path to the workspace to open in the test').string('workspacePath')
.describe('extensionDevelopmentPath', 'path to the extension to test').string('extensionDevelopmentPath')
.describe('extensionTestsPath', 'path to the extension tests').string('extensionTestsPath')
.describe('debug', 'do not run browsers headless').boolean('debug')
.describe('browser', 'browser in which integration tests should run').string('browser').default('browser', 'chromium')
.describe('help', 'show the help').alias('help', 'h');
@@ -26,22 +28,22 @@ if (optimist.argv.help) {
const width = 1200;
const height = 800;
async function runTestsInBrowser(browserType: string, endpoint: string): Promise<void> {
async function runTestsInBrowser(browserType: string, endpoint: url.UrlWithStringQuery, server: cp.ChildProcess): Promise<void> {
const browser = await playwright[browserType].launch({ headless: !Boolean(optimist.argv.debug) });
const page = (await browser.defaultContext().pages())[0];
await page.setViewport({ width, height });
const host = url.parse(endpoint).host;
const host = endpoint.host;
const protocol = 'vscode-remote';
const testWorkspaceUri = url.format({ pathname: path.resolve(optimist.argv.workspacePath), protocol, host, slashes: true });
const testExtensionUri = url.format({ pathname: path.resolve(optimist.argv.extensionDevelopmentPath), protocol, host, slashes: true });
const testFilesUri = url.format({ pathname: path.resolve(optimist.argv.extensionTestsPath), protocol, host, slashes: true });
const testWorkspaceUri = url.format({ pathname: URI.file(path.resolve(optimist.argv.workspacePath)).path, protocol, host, slashes: true });
const testExtensionUri = url.format({ pathname: URI.file(path.resolve(optimist.argv.extensionDevelopmentPath)).path, protocol, host, slashes: true });
const testFilesUri = url.format({ pathname: URI.file(path.resolve(optimist.argv.extensionTestsPath)).path, protocol, host, slashes: true });
const folderParam = testWorkspaceUri;
const payloadParam = `[["extensionDevelopmentPath","${testExtensionUri}"],["extensionTestsPath","${testFilesUri}"]]`;
await page.goto(`${endpoint}&folder=${folderParam}&payload=${payloadParam}`);
await page.goto(`${endpoint.href}&folder=${folderParam}&payload=${payloadParam}`);
await page.exposeFunction('codeAutomationLog', (type: string, args: any[]) => {
console[type](...args);
@@ -50,13 +52,30 @@ async function runTestsInBrowser(browserType: string, endpoint: string): Promise
page.on('console', async (msg: playwright.ConsoleMessage) => {
const msgText = msg.text();
if (msgText.indexOf('vscode:exit') >= 0) {
await browser.close();
try {
await browser.close();
} catch (error) {
console.error(`Error when closing browser: ${error}`);
}
try {
await pkill(server.pid);
} catch (error) {
console.error(`Error when killing server process tree: ${error}`);
}
process.exit(msgText === 'vscode:exit 0' ? 0 : 1);
}
});
}
async function launchServer(): Promise<string> {
function pkill(pid: number): Promise<void> {
return new Promise((c, e) => {
kill(pid, error => error ? e(error) : c());
});
}
async function launchServer(): Promise<{ endpoint: url.UrlWithStringQuery, server: cp.ChildProcess }> {
// Ensure a tmp user-data-dir is used for the tests
const tmpDir = tmp.dirSync({ prefix: 't' });
@@ -91,26 +110,23 @@ async function launchServer(): Promise<string> {
serverProcess?.stdout?.on('data', data => console.log(`Server stdout: ${data}`));
}
function teardownServer() {
if (serverProcess) {
serverProcess.kill();
}
}
process.on('exit', teardownServer);
process.on('SIGINT', teardownServer);
process.on('SIGTERM', teardownServer);
process.on('exit', () => serverProcess.kill());
process.on('SIGINT', () => serverProcess.kill());
process.on('SIGTERM', () => serverProcess.kill());
return new Promise(c => {
serverProcess?.stdout?.on('data', data => {
const matches = data.toString('ascii').match(/Web UI available at (.+)/);
if (matches !== null) {
c(matches[1]);
c({ endpoint: url.parse(matches[1]), server: serverProcess });
}
});
});
}
launchServer().then(async endpoint => {
return runTestsInBrowser(optimist.argv.browser, endpoint);
}, console.error);
launchServer().then(async ({ endpoint, server }) => {
return runTestsInBrowser(optimist.argv.browser, endpoint, server);
}, error => {
console.error(error);
process.exit(1);
});
test/integration/browser/yarn.lock
+10
View File
@@ -137,11 +137,21 @@ tmp@0.0.33:
dependencies:
os-tmpdir "~1.0.2"
tree-kill@1.2.2:
version "1.2.2"
resolved "https://registry.yarnpkg.com/tree-kill/-/tree-kill-1.2.2.tgz#4ca09a9092c88b73a7cdc5e8a01b507b0790a0cc"
integrity sha512-L0Orpi8qGpRG//Nd+H90vFB+3iHnue1zSSGmNOOCh1GLJ7rUKVwV2HvijphGQS2UmhUZewS9VgvxYIdgr+fG1A==
typescript@3.7.5:
version "3.7.5"
resolved "https://registry.yarnpkg.com/typescript/-/typescript-3.7.5.tgz#0692e21f65fd4108b9330238aac11dd2e177a1ae"
integrity sha512-/P5lkRXkWHNAbcJIiHPfRoKqyd7bsyCma1hZNUGfn20qm64T6ZBlrzprymeu918H+mB/0rIg2gGK/BXkhhYgBw==
vscode-uri@2.1.1:
version "2.1.1"
resolved "https://registry.yarnpkg.com/vscode-uri/-/vscode-uri-2.1.1.tgz#5aa1803391b6ebdd17d047f51365cf62c38f6e90"
integrity sha512-eY9jmGoEnVf8VE8xr5znSah7Qt1P/xsCdErz+g8HYZtJ7bZqKH5E3d+6oVNm1AC/c6IHUDokbmVXKOi4qPAC9A==
wrappy@1:
version "1.0.2"
resolved "https://registry.yarnpkg.com/wrappy/-/wrappy-1.0.2.tgz#b5243d8f3ec1aa35f1364605bc0d1036e30ab69f"
test/smoke/README.md
+2 -2
View File
@@ -13,13 +13,13 @@ yarn --cwd test/automation
yarn smoketest
# Dev (Web)
yarn smoketest --web --browser <chromium|firefox|webkit>
yarn smoketest --web --browser [chromium|firefox|webkit]
# Build (Electron)
yarn smoketest --build <path latest built version> --stable-build <path to previous stable version>
# Build (Web - read instructions below)
yarn smoketest --build <path to web server folder> --web --browser <chromium|firefox|webkit>
yarn smoketest --build <path to web server folder> --web --browser [chromium|firefox|webkit]
# Remote (Electron)
yarn smoketest --build <path latest built version> --remote
test/unit/README.md
+10 -16
View File
@@ -2,29 +2,23 @@
## Run (inside Electron)
The best way to run the unit tests is from the terminal. To make development changes to unit tests you need to be running `yarn watch`. See [Development Workflow](https://github.com/Microsoft/vscode/wiki/How-to-Contribute#incremental-build) for more details. From the `vscode` folder run:
./scripts/test.[sh|bat]
**OS X and Linux**
./scripts/test.sh
**Windows**
scripts\test
All unit tests are run inside a electron-browser environment which access to DOM and Nodejs api. This is the closest to the enviroment in which VS Code itself ships. Notes:
- use the `--debug` to see an electron window with dev tools which allows for debugging
- to run only a subset of tests use the `--run` or `--glob` options
## Run (inside browser)
You can run tests inside a browser instance too:
yarn test-browser --browser webkit --browser chromium
**OS X and Linux**
node test/unit/browser/index.js
**Windows**
node test\unit\browser\index.js
Unit tests from layers `common` and `browser` are run inside `chromium`, `webkit`, and (soon’ish) `firefox` (using playwright). This complements our electron-based unit test runner and adds more coverage of supported platforms. Notes:
- these tests are part of the continuous build, that means you might have test failures that only happen with webkit on _windows_ or _chromium_ on linux
- you can these tests locally via yarn `test-browser --browser chromium --browser webkit`
- to debug, open `<vscode>/test/unit/browser/renderer.html` inside a browser and use the `?m=<amd_module>`-query to specify what AMD module to load, e.g `file:///Users/jrieken/Code/vscode/test/unit/browser/renderer.html?m=vs/base/test/common/strings.test` runs all tests from `strings.test.ts`
- to run only a subset of tests use the `--run` or `--glob` options
## Run (with node)