10 Best JavaScript Crawler Libraries

List hand-picked by Openbase Experts
Learn More

simplecrawler

Flexible event driven crawler for node.

16.8K
2.1K
DefinitelyTyped
BSD-2-Clause
crawler

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

17.4K
6.3K
DefinitelyTyped
@algolia/netlify-plugin-crawler

Official Algolia Plugin for Netlify. Index your website to Algolia when deploying your project to Netlify with the Algolia Crawler

466
254
No support
MIT
pep

💯 Teach puppeteer new tricks through plugins.

164K
4.7K
Built-in
MIT
jc

Web crawler for Node.JS

506
233
No support
MIT
hcc

Distributed crawler powered by Headless Chrome

496
5.2K
No support
MIT
eh

Identifies bots/crawlers

88
1
Built-in
ISC
eb

Crawler(robots) decision middleware for Express

82
5
Built-in
MIT
express-spider-middleware

An ExpressJS middleware for detecting search engine crawlers and spiders, with the option of including a callback to perform additional search engine-specific logic such as logging or monitoring.

68
0
No support
MIT
spotlight

An object crawler/property search library.

43
141
No support
MIT
crawler-js

Opensource Framework Crawler in Node.js.

39
88
No support
ISC
lc

a simplified directed customizable website crawler

34
74
No support
MIT
sc

Crawl 100% JS single page apps with phantomjs and node.

6
15
No support
MIT
hc

A crawler implemented using a headless browser (Chrome).

4
12
No support
BSD-3-Clause
Description
Author
License
Type Definitions
ESM Support
Tree Shakeable
Popularity
Downloads/wk
GitHub Stars
Reviews
User Rating
Voted Good Docs
Voted Easy of Use
Voted Performant
Top Positive Review
Top Negative Review
Issues
Total Issues
Open Issues
Avg Time to Close
Pull Requests
Total Pull Requests
Open Pull Requests
Avg Time to Merge
Community
Maintainers
Contributors
Contributors with 3+ commits
Top Committers
GitHub Forks
Versions
# of Versions
Latest
Next
Beta
Newest
Oldest
Major Every
Minor Every
Patch Every
Dependencies
Dependencies
Dev Dependencies
Links
More Info
README
Tutorials
Install Command
There's Nothing Here
No packages are selected.
Select a few packages to compare.