SWE-Smith Expands to JavaScript with 6,099 Validated Bug Patches
SWE Smith Expands to JavaScript with 6,099 Validated Bug Patches The SWE Smith Multilingual project is extending SWE bench style task collection beyond Python. Its first expansi...
By Software Development Team
SWE-Smith Expands to JavaScript with 6,099 Validated Bug Patches
The SWE-Smith Multilingual project is extending SWE-bench-style task collection beyond Python. Its first expansion adds JavaScript support to SWE-Smith, producing and validating 6,099 synthetic bug patches across 74 popular JavaScript repositories.
The project’s planned work includes:
- Adding automatic repository installation and procedural modifications for JavaScript, TypeScript, Java, Rust, Golang, and C++.
- Running language-model-based bug-generation methods across the added repositories.
- Studying how programming languages and repositories affect model training dynamics and downstream coding-task performance.
- Exploring additional bug-generation strategies for SWE-Smith.
This report focuses on the JavaScript expansion.
Key Results
- 6,099 validated patches from 35,697 generated patches, a 17% validation rate
- 14 procedural modifiers for JavaScript bug generation
- An end-to-end Modal pipeline that reduces generation time from days to hours
- A Hugging Face dataset
What Is SWE-Smith?
SWE-Smith synthesizes realistic bugs in software repositories. It applies procedural modifications to working code, such as changing operators or swapping function arguments, to create defects that cause tests to fail. These patches can provide training data for code-repair models and tasks for evaluation benchmarks.
Previously, SWE-Smith supported only Python. The JavaScript expansion applies the same approach to JavaScript repositories.
Results by Repository
josdejong/mathjs produced the most validated patches. Its extensive arithmetic code and comprehensive unit tests provide conditions well suited to procedural bug generation. The repository produced 845 validated patches from 1,346 generated patches, for a 62.8% pass rate.
The other leading repositories were novnc/noVNC, Automattic/mongoose, bootstrap-vue/bootstrap-vue, and foliojs/pdfkit.
Highest-Producing Repositories
| Repository | Generated | Validated | Pass Rate |
|---|---|---|---|
| josdejong/mathjs | 1,346 | 845 | 62.8% |
| novnc/noVNC | 1,384 | 715 | 51.7% |
| Automattic/mongoose | 1,440 | 653 | 45.3% |
| bootstrap-vue/bootstrap-vue | 1,079 | 492 | 45.7% |
| foliojs/pdfkit | 797 | 408 | 51.2% |
All 74 Repositories
| Repository | Generated | Validated | Pass Rate |
|---|---|---|---|
| josdejong/mathjs | 1,346 | 845 | 62.8% |
| novnc/noVNC | 1,384 | 715 | 51.7% |
| Automattic/mongoose | 1,440 | 653 | 45.3% |
| bootstrap-vue/bootstrap-vue | 1,079 | 492 | 45.7% |
| foliojs/pdfkit | 797 | 408 | 51.2% |
| bpampuch/pdfmake | 1,961 | 264 | 13.5% |
| caolan/async | 645 | 233 | 36.3% |
| enzymejs/enzyme | 819 | 228 | 30.6% |
| Netflix/falcor | 1,046 | 166 | 15.9% |
| nock/nock | 337 | 161 | 47.8% |
| redux-saga/redux-saga | 277 | 153 | 55.4% |
| sveltejs/svelte | 274 | 151 | 55.1% |
| necolas/react-native-web | 506 | 134 | 26.5% |
| axios/axios | 304 | 133 | 43.8% |
| websockets/ws | 384 | 131 | 35.2% |
| svg/svgo | 171 | 126 | 73.7% |
| nightwatchjs/nightwatch | 963 | 121 | 12.6% |
| emotion-js/emotion | 210 | 113 | 53.8% |
| josdejong/jsoneditor | 920 | 111 | 12.1% |
| mochajs/mocha | 402 | 108 | 26.9% |
| iamkun/dayjs | 173 | 99 | 57.6% |
| brianc/node-postgres | 319 | 97 | 30.4% |
| Shopify/draggable | 222 | 90 | 40.5% |
| welldone-software/why-did-you-render | 162 | 66 | 40.7% |
| balderdashy/sails | 157 | 61 | 38.9% |
| reactjs/react-transition-group | 95 | 49 | 51.6% |
| remy/nodemon | 212 | 47 | 22.2% |
| advplyr/audiobookshelf | 1,590 | 45 | 2.8% |
| segmentio/evergreen | 114 | 41 | 36.0% |
| webpack/webpack | 303 | 31 | 10.2% |
| jantimon/html-webpack-plugin | 143 | 13 | 9.1% |
| babel/babel | 78 | 9 | 11.7% |
| mholt/PapaParse | 422 | 3 | 0.7% |
| marko-js/marko | 255 | 2 | 0.8% |
| 11ty/eleventy | 1,032 | 0 | 0.0% |
| GoogleChrome/workbox | 88 | 0 | 0.0% |
| HabitRPG/habitica | 284 | 0 | 0.0% |
| Modernizr/Modernizr | 225 | 0 | 0.0% |
| Qix-/color | 35 | 0 | 0.0% |
| Unitech/pm2 | 898 | 0 | 0.0% |
| akiran/react-slick | 40 | 0 | 0.0% |
| davila7/claude-code-templates | 1,357 | 0 | 0.0% |
| diegomura/react-pdf | 631 | 0 | 0.0% |
| elbywan/wretch | 7 | 0 | 0.0% |
| expressjs/express | 31 | 0 | 0.0% |
| expressjs/multer | 44 | 0 | 0.0% |
| facebookexperimental/Recoil | 178 | 0 | 0.0% |
| forwardemail/superagent | 68 | 0 | 0.0% |
| forwardemail/supertest | 3 | 0 | 0.0% |
| gka/chroma.js | 33 | 0 | 0.0% |
| hakimel/reveal.js | 1,965 | 0 | 0.0% |
| handsontable/handsontable | 502 | 0 | 0.0% |
| hapijs/joi | 650 | 0 | 0.0% |
| highlightjs/highlight.js | 759 | 0 | 0.0% |
| immutable-js/immutable-js | 636 | 0 | 0.0% |
| impress/impress.js | 114 | 0 | 0.0% |
| jashkenas/backbone | 7 | 0 | 0.0% |
| jquery/jquery | 454 | 0 | 0.0% |
| koajs/koa | 97 | 0 | 0.0% |
| kriskowal/q | 128 | 0 | 0.0% |
| layui/layui | 105 | 0 | 0.0% |
| louislam/uptime-kuma | 1,227 | 0 | 0.0% |
| mdx-js/mdx | 242 | 0 | 0.0% |
| mrdoob/three.js | 0 | 0 | 0.0% |
| mui/material-ui | 113 | 0 | 0.0% |
| parallax/jsPDF | 85 | 0 | 0.0% |
| piskelapp/piskel | 1,131 | 0 | 0.0% |
| pqina/filepond | 6 | 0 | 0.0% |
| remarkjs/react-markdown | 43 | 0 | 0.0% |
| serverless/serverless | 1,144 | 0 | 0.0% |
| sql-js/sql.js | 8 | 0 | 0.0% |
| tj/commander.js | 321 | 0 | 0.0% |
| usebruno/bruno | 962 | 0 | 0.0% |
| webtorrent/webtorrent | 534 | 0 | 0.0% |
Half of the 74 repositories produced no validated patches:
- 20 repositories failed during pre-gold validation because test results were not parsed correctly.
- 20 repositories produced too few patches, fewer than 50, making validation costs outweigh their contribution.
Modifier Effectiveness
The JavaScript implementation includes 14 procedural modifiers. Their results varied considerably.
Leading Modifiers
| Modifier | Generated | Validated | Pass Rate |
|---|---|---|---|
func_pm_op_flip | 5,380 | 1,117 | 30.8% |
func_pm_arg_swap | 3,228 | 722 | 32.7% |
func_pm_remove_assign | 3,602 | 632 | 25.4% |
func_pm_remove_cond | 4,009 | 604 | 21.9% |
func_pm_ctrl_invert_if | 1,890 | 490 | 35.7% |
The func_pm_op_flip modifier generated the most validated patches, with 1,117, nearly twice as many as the next most productive modifier. It changes binary operators, including === to !==, < to >=, and && to ||, allowing it to apply to many expressions across the repositories.
All 14 Modifiers
| Modifier | Description | Generated | Validated | Pass Rate |
|---|---|---|---|---|
func_pm_op_flip | Flip binary operators | 5,380 | 1,117 | 30.8% |
func_pm_arg_swap | Swap function arguments | 3,228 | 722 | 32.7% |
func_pm_remove_assign | Remove assignments | 3,602 | 632 | 25.4% |
func_pm_remove_cond | Remove conditionals | 4,009 | 604 | 21.9% |
func_pm_ctrl_invert_if | Swap if/else branches | 1,890 | 490 | 35.7% |
func_pm_op_change_const | Modify numeric constants | 2,559 | 487 | 28.6% |
func_pm_op_swap | Swap operands | 4,945 | 443 | 13.2% |
func_pm_op_change | Change operators (+, -, *, /) | 2,157 | 336 | 25.4% |
func_pm_op_break_chains | Break chained operations | 2,343 | 304 | 18.9% |
func_pm_ternary_swap | Swap ternary branches | 1,442 | 277 | 27.6% |
func_pm_remove_loop | Remove loop constructs | 1,356 | 253 | 24.6% |
func_pm_remove_ternary | Remove ternary expressions | 1,335 | 185 | 20.4% |
func_pm_aug_assign_swap | Swap augmented assignments | 1,063 | 143 | 18.6% |
func_pm_ctrl_shuffle | Shuffle statements in loops | 388 | 106 | 36.7% |
Scaling with Modal
Previously, large-scale bug generation required days of local computation. The complete pipeline now runs on Modal, reducing the process to hours.
The pipeline performs three stages:
- Generate patches: Apply procedural modifiers to repository code.
- Validate patches: Run each repository’s test suite in isolated containers.
- Store results: Save the outputs in the SWE-smith-js Hugging Face dataset.
Creating Tasks from Pull Requests with SWE-gen
In addition to synthetic patches, task creation is being expanded through SWE-gen. The tool converts merged GitHub pull requests into verifiable tasks. It supports multiple languages by detecting the language, build system, and test framework. SWE-gen reverses a pull request to reconstruct the buggy state, then verifies agent behavior with fail-to-pass tests.
SWE-gen-JS contains 1,000 tasks generated from 30 popular open-source JavaScript and TypeScript repositories. These tasks are derived from real bug fixes and complement synthetic tasks by representing multi-file changes and test configurations that procedural modifiers cannot always reproduce.
Next Steps
The project is improving its repository-construction agent to increase the number of productive repositories. Planned changes include:
- Pre-validation checks: Require more than zero pre-gold tests to pass before adding a profile.
- Smarter repository selection: Use the 20 unproductive samples to improve selection heuristics.
The func_pm_ctrl_shuffle modifier is also identified as an area for improvement. Its HAS_LOOP filter may be too restrictive, limiting where the modifier can be applied.