Ulugbek Abdullaev
02629fd4b7
* nes-datagen: stream large JSON array inputs to avoid 2 GiB readFile limit Reading the whole input via fs.readFile fails for files larger than 2 GiB (and exceeds V8's max string length). Add a streaming JSON-array parser and use it in both the sequential and parallel pipeline paths so multi-GB recordings can be processed with bounded memory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * nes-datagen: also accept JSON Lines (NDJSON) input Auto-detect the input format from the first non-whitespace character: a leading '[' is parsed as a single JSON array, otherwise the file is parsed as JSON Lines (one JSON object per line). Both formats are streamed so multi-GB inputs work regardless of shape. Rename streamJsonArray -> streamJsonRecords to reflect the broader purpose. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * nes-datagen: infer JSON vs JSON Lines input format from file extension Use the file extension (.jsonl/.ndjson -> JSON Lines, otherwise JSON array) to select the streaming parser instead of sniffing the content. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * nes-datagen: validate streamed JSON arrays for truncation and malformed input The new streaming parser previously accepted any prefix of a JSON array silently: a truncated file (no closing ']'), a missing element between commas, a trailing comma or trailing data after the array all produced zero or fewer records rather than an error. That is especially dangerous for the multi-GB inputs this parser was introduced for, because the underlying file is much more likely to be incomplete. Tighten the state machine to surface these as errors, matching what the old whole-file JSON.parse would have done, and add tests for each case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * nes-datagen: stream worker-result merging and final output write For multi-GB inputs the parent process was still hitting V8's ~512 MiB max-string-length limit in two places after the input-side fix: 1. Merging worker result files used fs.promises.readFile + JSON.parse on each result, but with a 5+ GB input split N ways each per-worker file is hundreds of MB of similar-shaped data and easily exceeds the string limit. 2. writeSamples serialized the entire validSamples array via a single JSON.stringify(arr, null, 2) before writing, which has the same problem on output. Switch both to stream over individual records: - A new shared openWriteStream(filePath) helper wraps fs.createWriteStream, attaches an 'error' listener immediately (so async write failures don't surface as uncaughtException and skip cleanup), awaits backpressure via the per-write callback, and exposes an idempotent close(). - writeChunkFiles uses the helper inside a try/finally so any mid-stream ENOSPC/EIO bubbles up cleanly and the tmp dir is still removed. - The merge step now uses streamJsonRecords<ISample>(resultPath), so the parent never materializes a single worker's output as one string. - writeSamples emits the output JSON array incrementally: per-sample JSON.stringify(..., null, 2) (indented two spaces to match the previous layout) joined with ',\n'. Byte size is accumulated for the existing IWriteResult.fileSize. Also documents that single-process loadAndParseInput still buffers the full row set in memory and that --parallelism is required for very large inputs (workers each only load their slice). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * nes-datagen: switch output to JSON Lines Both the final user-facing output and the per-worker intermediate result files now use JSON Lines (one record per line) instead of a pretty-printed JSON array. JSONL is dramatically simpler to write and read incrementally: no surrounding brackets/commas to track, no multi-line per-element indentation, just JSON.stringify + '\n' per record on the write side and split-on-newline + JSON.parse per non-empty line on the read side (this is what streamJsonRecords already does when it detects the .jsonl extension). Changes: - writeSamples emits one JSON.stringify(sample) + '\n' per validated sample via no array wrapper, no pretty-printing.openWriteStream - resolveOutputPath defaults the implicit output to <input>_output.jsonl (was <input>_output.json). - Per-worker result files in runInputPipelineParallel are now result_${w}.jsonl, so the merge step's streamJsonRecords auto-picks the JSONL parser from the extension. - E2e tests updated to read JSONL (split on newline, JSON.parse per line) and to use .jsonl output paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * nes-datagen: surface worker-result parse errors and update --out help text Two review follow-ups: 1. The merge step that streams each worker's result file used to wrap the iteration in a try/catch that downgraded any parse error to a console.error warning. With the new streaming reader that is unsafe: streamJsonRecords yields N valid records first and then throws on a malformed/truncated tail, leaving those N partial records already in allSamples. The pipeline would then quietly emit a truncated training-data output. Drop the swallowing try/catch so a corrupt worker result aborts the run non-zero. 2. The --out help text in simulationOptions.ts still advertised the old JSON-array default (<input>_output.json). Update it to reflect the JSONL output, and also note in --input that the format is inferred from the .jsonl/.ndjson extension. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
GitHub Copilot - Your autonomous AI peer programmer
GitHub Copilot is an AI peer programming tool that transforms how you write code in Visual Studio Code.
GitHub Copilot agents handle complete coding tasks end-to-end, autonomously planning work, editing files, running commands, and self-correcting when they hit errors. You can also leverage inline suggestions for quick coding assistance and inline chat for precise, focused edits directly in the editor.
Sign up for GitHub Copilot Free!
Getting access to GitHub Copilot
Sign up for GitHub Copilot Free, or request access from your enterprise admin.
To access GitHub Copilot, an active GitHub Copilot subscription is required. You can read more about our business and individual offerings at github.com/features/copilot.
Build with autonomous agents
Let AI agents implement complex features end-to-end. Give an agent a high-level task and it breaks the work into steps, edits multiple files, runs terminal commands, and self-corrects when it hits errors or failing tests. Agents excel at building new features, debugging and fixing failing tests, refactoring codebases, and collaborating via pull requests.
Manage sessions from a central view. Run multiple agent sessions in parallel and track them in one place. Monitor session status, switch between active work, review file changes, and resume where you left off.
Run agents with your preferred harness. Use agents locally in VS Code, in the background via Copilot CLI, or Cloud via Copilot Coding Agent. You can also work with providers like Claude and Codex, and hand tasks off between agent types with context preserved all within the VS Code.
Use agents to plan before you build with the Plan agent, which breaks tasks into structured implementation plans and asks clarifying questions. When your plan is ready, hand it off to an implementation agent to execute it. You can also delegate tasks to cloud agents that create branches, implement changes, and open pull requests for your team to review.
More ways to code with AI
Receive intelligent inline suggestions as you type with ghost text suggestions and next edit suggestions, helping you write code faster. Copilot predicts your next logical change, and you can accept suggestions with the Tab key.
Use inline chat for targeted edits by pressing Ctrl+I/Cmd+I to open a chat prompt directly in the editor. Describe a change and Copilot suggests edits in place for refactoring methods, adding error handling, or explaining complex algorithms without leaving your editor.
Customize AI for your workflow
Agents work best when they understand your project's conventions and have the right tools. Tailor Copilot so it generates code that fits your codebase from the start.
Project context. Use custom instructions to specify project-wide or task-specific context and coding guidelines.
Add specialized capabilities. Teach Copilot specialized capabilities with agent skills or define specialized personas with custom agents.
Connect to external tools and services. Extend agents further with tools from MCP servers and extensions to give Copilot a gateway to external data sources, APIs, or specialized tools.
Supported languages and frameworks
GitHub Copilot works on any language, including Java, PHP, Python, JavaScript, Ruby, Go, C#, or C++. Because it’s been trained on languages in public repositories, it works for most popular languages, libraries and frameworks.
Version compatibility
As Copilot Chat releases in lockstep with VS Code due to its deep UI integration, every new version of Copilot Chat is only compatible with the latest and newest release of VS Code. This means that if you are using an older version of VS Code, you will not be able to use the latest Copilot Chat.
Only the latest Copilot Chat versions will use the latest models provided by the Copilot service, as even minor model upgrades require prompt changes and fixes in the extension.
Privacy and preview terms
By using Copilot Chat you agree to GitHub Copilot chat preview terms. Review the transparency note to understand about usage, limitations and ways to improve Copilot Chat during the technical preview.
Please refer to our Privacy Statement to learn about the data we collect, how we use it, and the controls available to you.
To get the latest security fixes, please use the latest version of the Copilot extension and VS Code.
Resources & next steps
- Sign up for GitHub Copilot Free: Explore Copilot's AI capabilities at no cost before upgrading to a paid plan.
- If you're using Copilot for your business, check out Copilot Business and Copilot Enterprise.
- Copilot Quickstart: Discover the key features of Copilot in VS Code.
- Agents Tutorial: Get started with autonomous agents across different environments.
- VS Code on YouTube: Watch the latest demos and updates on the VS Code channel.
- Frequently Asked Questions: Get answers to commonly asked questions about Copilot in VS Code.
- Provide Feedback: Send us your feedback and feature request to help us make GitHub Copilot better!
Data and telemetry
The GitHub Copilot Extension for Visual Studio Code collects usage data and sends it to Microsoft to help improve our products and services. Read our privacy statement to learn more. This extension respects the telemetry.telemetryLevel setting which you can learn more about at https://code.visualstudio.com/docs/supporting/faq#_how-to-disable-telemetry-reporting.
Trademarks
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.
License
Copyright (c) Microsoft Corporation. All rights reserved.
Licensed under the MIT license.