sru

scrapy-random-useragent

Scrapy Middleware to set a random User-Agent for every Request.

Showing:

Popularity

Downloads/wk

0

GitHub Stars

185

Maintenance

Last Commit

5yrs ago

Contributors

3

Package

Dependencies

0

License

MIT

Categories

Readme

Scrapy Random User-Agent

Does your scrapy spider get identified and blocked by servers because you use the default user-agent or a generic one?

Use this random_useragent module and set a random user-agent for every request. You are limited only by the number of different user-agents you set in a text file.

Installing

Installing it is pretty simple.

.. code-block:: python

pip install scrapy-random-useragent

Usage

In your settings.py file, update the DOWNLOADER_MIDDLEWARES variable like this.

.. code-block:: python

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
    'random_useragent.RandomUserAgentMiddleware': 400
}

This disables the default UserAgentMiddleware and enables the RandomUserAgentMiddleware.

Then, create a new variable USER_AGENT_LIST with the path to your text file which has the list of all user-agents (one user-agent per line).

.. code-block:: python

USER_AGENT_LIST = "/path/to/useragents.txt"

Now all the requests from your crawler will have a random user-agent picked from the text file.

Rate & Review

Great Documentation0
Easy to Use0
Performant0
Highly Customizable0
Bleeding Edge0
Responsive Maintainers0
Poor Documentation0
Hard to Use0
Slow0
Buggy0
Abandoned0
Unwelcoming Community0
100