10 Best JavaScript Crawler Libraries

List hand-picked by Openbase Experts
Learn More

crawler

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

License Icon
License: Unknown
TypeScript Icon
TypeScript Definitions: DefinitelyTyped
User Rating
4.0/ 5
5
Top Feedback
3Great Documentation
1Easy to Use
1Performant
GitHub Stars
6K
Weekly Downloads
8K
Last Commit
2mos ago

simplecrawler

Flexible event driven crawler for node.

License Icon
License: BSD-2-Clause
TypeScript Icon
TypeScript Definitions: DefinitelyTyped
User Rating
4.3/ 5
3
Top Feedback
2Great Documentation
1Easy to Use
GitHub Stars
2K
Weekly Downloads
10K
Last Commit
10mos ago
pep

puppeteer-extra-plugin-stealth

💯 Teach puppeteer new tricks through plugins.

License Icon
License: MIT
TypeScript Icon
TypeScript Definitions: Built-In
User RatingN/A
Top Feedback
1Great Documentation
GitHub Stars
3K
Weekly Downloads
72K
Last Commit
1mo ago

@opd/crawler

Web crawler based on Puppeteer

License Icon
License: MIT
TypeScript Icon
TypeScript Definitions: Not Found
User RatingN/A
Top Feedback
N/A
GitHub Stars
11
Weekly Downloads
5
Last Commit
1mo ago
hcc

headless-chrome-crawler

Distributed crawler powered by Headless Chrome

License Icon
License: MIT
TypeScript Icon
TypeScript Definitions: DefinitelyTyped
User RatingN/A
Top Feedback
N/A
GitHub Stars
5K
Weekly Downloads
676
Last Commit
1yr ago
eb

express-bot

Crawler(robots) decision middleware for Express

License Icon
License: MIT
TypeScript Icon
TypeScript Definitions: Built-In
User RatingN/A
Top Feedback
N/A
GitHub Stars
5
Weekly Downloads
11
Last Commit
6mos ago
eh

express-hawk

Identifies bots/crawlers

License Icon
License: ISC
TypeScript Icon
TypeScript Definitions: Built-In
User RatingN/A
Top Feedback
N/A
GitHub Stars
1
Weekly Downloads
4
Last Commit
8mos ago
jc

js-crawler

Web crawler for Node.JS

License Icon
License: MIT
TypeScript Icon
TypeScript Definitions: DefinitelyTyped
User RatingN/A
Top Feedback
N/A
GitHub Stars
231
Weekly Downloads
76
Last Commit
4yrs ago

crawler-js

Opensource Framework Crawler in Node.js.

License Icon
License: ISC
TypeScript Icon
TypeScript Definitions: DefinitelyTyped
User RatingN/A
Top Feedback
N/A
GitHub Stars
88
Weekly Downloads
40
Last Commit
3yrs ago
lc

light-crawler

a simplified directed customizable website crawler

License Icon
License: MIT
TypeScript Icon
TypeScript Definitions: DefinitelyTyped
User RatingN/A
Top Feedback
N/A
GitHub Stars
74
Weekly Downloads
33
Last Commit
1yr ago

spotlight

An object crawler/property search library.

License Icon
License: MIT
TypeScript Icon
TypeScript Definitions: DefinitelyTyped
User RatingN/A
Top Feedback
N/A
GitHub Stars
139
Weekly Downloads
10
Last Commit
4yrs ago

express-spider-middleware

An ExpressJS middleware for detecting search engine crawlers and spiders, with the option of including a callback to perform additional search engine-specific logic such as logging or monitoring.

License Icon
License: MIT
TypeScript Icon
TypeScript Definitions: DefinitelyTyped
User RatingN/A
Top Feedback
N/A
GitHub Stars
N/A
Weekly Downloads
7
Last Commit
2yrs ago
sc

spa-crawler

Crawl 100% JS single page apps with phantomjs and node.

License Icon
License: MIT
TypeScript Icon
TypeScript Definitions: DefinitelyTyped
User RatingN/A
Top Feedback
N/A
GitHub Stars
12
Weekly Downloads
7
Last Commit
3yrs ago
hc

headless-crawler

A crawler implemented using a headless browser (Chrome).

License Icon
License: BSD-3-Clause
TypeScript Icon
TypeScript Definitions: DefinitelyTyped
User RatingN/A
Top Feedback
N/A
GitHub Stars
12
Weekly Downloads
2
Last Commit
3yrs ago