npm i eslint-remote-tester


CLI tool for testing given ESlint rules against multiple repositories at once.

by Ari Perkkiö

3.0.0 (see all)License:MITTypeScript:Built-In
npm i eslint-remote-tester


eslint-config-airbnb eslint-plugin-cypress eslint-plugin-import eslint-plugin-jest eslint-plugin-jest-dom eslint-plugin-jsx-a11y eslint-plugin-lit eslint-plugin-mocha eslint-plugin-next eslint-plugin-node eslint-plugin-react eslint-plugin-react-hooks eslint-plugin-react-redux eslint-plugin-regexp eslint-plugin-sonarjs eslint-plugin-testing-library eslint-plugin-unicorn typescript-eslint-eslint-plugin eslint-core eslint-core-ts


Installation | Configuration | How and when to use | Github CI Actions | Examples

eslint-remote-tester is a CLI tool for testing given ESlint rules against multiple repositories at once. It's designed to be used when validating regression of new rules. It can be used to spot whether a new rule flags false positives or crashes ESLint completely. CIs can be configured to verify regression of large set of rules so that possible null pointers or any unexpected errors are caught immediately.


# In your project directory
$ yarn add --dev eslint-remote-tester

# eslint is also required as peer dependency
$ yarn add --dev eslint

The underlying git integration is done via simple-git. It requires git to be installed and that it can be called using the command git.



Add new script to your package.json file.

"scripts": {
    "lint:remote": "eslint-remote-tester",


Create new configuration file eslint-remote-tester.config.js in the root of your project. This is used as configuration file by default. Use -c or --config argument for custom configuration file, e.g. --config path/custom.config.js.

/** @type {import('eslint-remote-tester').Config} */
module.exports = {
    repositories: ['mui-org/material-ui', 'reach/reach-ui'],
    extensions: ['js', 'jsx', 'ts', 'tsx'],
    eslintrc: {
        root: true,
        extends: ['eslint:recommended'],
    pathIgnorePattern: `(${[
        '\\/\\.', // Any file or directory starting with dot, e.g. ".git"

Configuration file can also be written in TypeScript if ts-node is installed. Use --config argument for TypeScript configuration file, e.g. --config eslint-remote-tester.config.ts.

import type { Config } from 'eslint-remote-tester';

const config: Config = {
    repositories: ['mui-org/material-ui', 'reach/reach-ui'],
    extensions: ['js', 'jsx', 'ts', 'tsx'],
    eslintrc: {
        root: true,
        extends: ['eslint:recommended'],

export default config;

Configuration options

NameDescription                              TypeRequiredDefaultExample
repositoriesRepositories to scan in format of owner/project. See eslint-remote-tester-repositories for shared list of repositories.string[]['mui-org/material-ui', 'reach/reach-ui']
extensionsExtensions of files under scanningstring[]['js', 'jsx', 'ts', 'tsx']
eslintrcESLint configuration. Supports lazy initialization based on currently tested repository when a function is passed. Function is called with current repository and its location on filesystem.object \| ({ location, repository }) => object See Configuring ESLint{ root: true, extends: ['eslint:all'] } (options) => options.repository === 'my-repo' ? ({ extends: ['eslint:all'] }) : ({})
pathIgnorePatternRegexp pattern string used to exclude pathsstring(node_modules\|docs\|\\/\\.git)
maxFileSizeBytesMax file size used to exclude bigger filesnumber20000001500000
rulesUnderTestingArray of rules or a filter method used to filter out results. Use undefined or empty array when ESLint crashes are the only interest. Filter method is called with ruleId and options.string[] \| (ruleId, { repository }) => boolean[]['no-empty', 'react/sort-prop-types'] (ruleId, options) => ruleId === 'no-undef' && options.repository === 'owner/repo'
resultParserSyntax for the result parserplaintext\|markdownmarkdown on CLI. plaintext on CImarkdown
concurrentTasksMaximum amount of tasks run concurrentlynumber53
CIFlag used to set CI mode. process.env.CI is used when not set.booleanvalue of process.env.CI === 'true'true
logLevelFilter log messages based on their priorityverbose\|info\|warn\|errorverbosewarn
slowLintTimeLimitTime limit before linting of a single file is considered as slow, and logged as warning. Disabled by default.number\|null5 // 5 seconds
cacheFlag used to enable caching of cloned repositories. For CIs it's ideal to disable caching due to limited disk space.booleantruetrue
timeLimitTime limit before scan is interrupted and exited successfully. Ideal for avoiding CI timeouts in regression tests.number5.5 * 60 * 60, // 5 hours 30 minutes5 * 60 * 60 // 5 hours
compareFlag used to enable result comparison mode. Compares results of two scans and output the diff. Ideal for identifying new false positives when fixing existing rules. See Fixing existing rules.booleanfalsetrue
updateComparisonReferenceFlag used to enable result comparison reference updating. Indicates whether comparison base should be updated after scan has finished. Ideal to be turned off once initial comparison base has been collected.booleantruetrue
onCompleteCallback invoked once scan is completed. Asynchronous functions are supported. Ideal for extending the process with custom features.(results, comparisonResults, repositoryCount) => void|Promise<void>. See onComplete example.async (results, comparisonResults, repositoryCount) => {}


Run yarn lint:remote. Results are written into ./eslint-remote-tester-results folder.

How and when to use

Disclaimer: This section is still work in progress.

Developing new ESLint rules

When developing new ESLint rules you'll be most interested in results reported by the rule. As the AST of Javascript and Typescript can cause very unexpected results it is not enough to test the rule only against unit tests and a small amount of repositories. Testing the rule against 1000's of repositories can reveal unexpected results:

  • Does the rule report the intended patterns?
  • Does the rule falsely mark valid patterns as errors?
  • Are the error messages set as expected?

Follow these steps to test the new rule easily:

  1. Include new rule in rulesUnderTesting
  2. Configure eslintrc with the new rule and its options
  3. Run eslint-remote-tester locally
  4. Check generated results at ./eslint-remote-tester-results

Fixing existing rules

Fixing existing rules without breaking old functionality or introducing new bugs can be difficult. By testing the rule against 1000's of repositories and comparing these reports against previous build's reports it is easily seen whether anything broke.

Running eslint-remote-tester in comparison mode allows developers to see the exact changes in ESLint reports their code changes introduced. Example of comparison results can be found at Examples section.

Follow these steps to test code changes of existing rules easily:

  1. Include given rule in rulesUnderTesting
  2. Configure eslintrc with the rule and its options
  3. Stash code changes of the fix so that the current setup matches the latest build (typically master branch)
  4. Enable compare in the configuration file. This will generate comparison reference results.
  5. Run eslint-remote-tester locally
  6. Apply code changes of the fix back
  7. Disable updateComparisonReference in the configuration file. This will disable comparison reference updates - meaning that all next comparison runs are compared against the previously generated results (e.g. master branch).
  8. Run eslint-remote-tester locally again
  9. Check generated results at ./eslint-remote-tester-results/comparison-results. New reports introduced by these code changes are saved in Reports not seen with latest changes are saved in
  10. Repeat steps 8-9 and keep fixing the rule until does not contain new false positives and does not contain unintended reports.

Plugin maintainer making sure new PRs don't introduce new false positives

Reviewing PRs of ESLint plugin projects can be slow if maintainers are expected to test every change manually. Plugin projects can set their CI to run eslint-remote-tester on comparison mode for each new PR. The CI will compare ESLint reports between the master and PR branches, and report all new and removed reports. This way the reviewers can easily see whether PR actually fixed the issue and if it introduced new bugs.

Projects using GitHub CIs can utilize eslint-remote-tester-compare-action for setting up PR comparisons.

Plugin maintainer making sure all existing rules do not crash

Plugin projects can use eslint-remote-tester for smoke testing. All the rules of given project can be run against 1000's of repositories so that only crashing rules are detected. The job can even be configured to run for a limited time so that it will test as many repositories as it can in a specific time. This can be done by setting timeLimit configuration. For Github CI the maximum time limit can be as long as 6 hours.

This project is set to run 6 hour long weekly scheduled tests using some of the most well-known ESLint community plugins. Typically the fastest plugins can test more than 10K repositories in the 6 hour time limit.

Projects using GitHub CIs can utilize eslint-remote-tester-run-action for setting up smoke tests.

Configuration maintainer making sure all repositories follow the rules

Configuration maintainers (eslint-config-*) can check all their repositories follow rules of the configuration. By using an all-inclusive filter in rulesUnderTesting all ESLint reports are picked, e.g. rulesUnderTesting: () => true.

The arguments of filter callback can be used to exclude specific rules from specific repositories:

rulesUnderTesting: function ruleFilter(ruleId, options) {
    if (ruleId === 'node/no-process-env' && options.repository === 'username/my-cli-package') {
        // my-cli-package is allowed to use process.env
        return false;

    // All other rules & repositories
    return true;

Github CI Actions



Comparison results:



For easiest setup eslint-remote-tester-repositories npm package is provided. See documentation.

Target repositories can easily be found using They provide an API for querying repositories which depend on certain project. Example usage at repository-query-script.

No alternatives found
No tutorials found
Add a tutorial
No dependencies found

Rate & Review

No reviews found
Be the first to rate