cre
creepy
pypi i creepy
cre

creepy

Dead simple web crawler for Python

by Wei-Ning Huang

0.1.6 (see all)License:GPL
pypi i creepy
Readme

PyPI Version PyPI Download

Creepy

Dead simple web crawler for Python

There are already a lot of web crawlers for Python, such as Scrapy. Creepy is yet another web crawler for Python, which ains to provide a simple and light way to write web crawlers.

Example usage

from creepy import Crawler

class MyCrawler(Crawler):
def process_document(self, doc):
    if doc.status == 200:
        print '[%d] %s' % (doc.status, doc.url)
    # Do something with doc.text (the content of the page)
else:
    pass

crawler = MyCrawler()
crawler.set_follow_mode(Crawler.F_SAME_HOST)
crawler.add_url_filter('\.(jpg|jpeg|gif|png|js|css|swf)$')
crawler.crawl('http://www.example.com/')

Installation

  1. Install from PyPI:
    pip install creepy
  2. Arch Linux users can find it on AUR or using Yaourt:
    yaourt -S python2-creepy-git

Bugs

  • Please report bugs to the github issure tracker.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

GitHub Stars

40

LAST COMMIT

6yrs ago

MAINTAINERS

1

CONTRIBUTORS

6

OPEN ISSUES

0

OPEN PRs

2
VersionTagPublished
0.1.6
6yrs ago
0.1.5
8yrs ago
0.1.1
9yrs ago
0.1.0
9yrs ago
No alternatives found
No tutorials found
Add a tutorial