reppy

Replaces the built-in robotsparser with a RFC-conformant implementation that supports modern robots.txt constructs like Sitemaps, Allow, and Crawl-delay. Main features:

• Memoization of fetched robots.txt
• Expiration taken from the Expires header
• Batch queries
• Configurable user agent for fetching robots.txt
• Automatic refetching based on expiration