reppy2
pypi i reppy2

reppy2

Modern robots.txt Parser for Python

by seomoz

0.3.6 (see all)License:MIT License
pypi i reppy2
Readme

Replaces the built-in robotsparser with a RFC-conformant implementation that supports modern robots.txt constructs like Sitemaps, Allow, and Crawl-delay. Main features:

  • Memoization of fetched robots.txt
  • Expiration taken from the Expires header
  • Batch queries
  • Configurable user agent for fetching robots.txt
  • Automatic refetching basing on expiration

This is a patched fork of the last pure Python version that works on Python 2 and 3.

GitHub Stars

172

LAST COMMIT

2yrs ago

MAINTAINERS

1

CONTRIBUTORS

21

OPEN ISSUES

17

OPEN PRs

7
VersionTagPublished
0.3.6
10mos ago
0.3.5
2yrs ago
No alternatives found
No tutorials found
Add a tutorial