pro
proxypy
pypi i proxypy
pro

proxypy

Proxy collector

by DevAlone

2.0 (see all)
pypi i proxypy
Readme

proxy_py

proxy_py is a program which collects proxies, saves them in a database and makes periodically checks. It has a server for getting proxies with nice API(see below).

Where is the documentation?

It's here

How to build?

1 Clone this repository

git clone https://github.com/DevAlone/proxy_py.git

2 Install requirements

cd proxy_py
pip3 install -r requirements.txt

3 Create settings file

cp config_examples/settings.py proxy_py/settings.py

4 Install postgresql and change database configuration in settings.py file

5 (Optional) Configure alembic

6 Run your application

python3 main.py

7 Enjoy!

I'm too lazy. Can I just use it?

TODO: update, old version!

Yes, you can download virtualbox image here. After downloading check that port forwarding is still working, you need forwarding of 55555 host port to 55555 guest.

How to get proxies?

proxy_py has a server, based on aiohttp, which is listening 127.0.0.1:55555 (you can change it in the settings file) and provides proxies. To get proxies you should send the following json request on address http://127.0.0.1:55555/api/v1/ (or other domain if behind reverse proxy):

{
    "model": "proxy",
    "method": "get",
    "order_by": "response_time, uptime"
}

Note: order_by makes the result sorted by one or more fields(separated by comma). You can skip it. The required fields are model and method.

It's gonna return you the json response like this:

{
    "count": 1,
    "data": [{
            "address": "http://127.0.0.1:8080",
            "auth_data": "",
            "bad_proxy": false,
            "domain": "127.0.0.1",
            "last_check_time": 1509466165,
            "number_of_bad_checks": 0,
            "port": 8080,
            "protocol": "http",
            "response_time": 461691,
            "uptime": 1509460949
        }
    ],
    "has_more": false,
    "status": "ok",
    "status_code": 200
}

Note: All fields except protocol, domain, port, auth_data, checking_period and address CAN be null

Or error if something went wrong:

{
    "error_message": "You should specify \"model\"",
    "status": "error",
    "status_code": 400
}

Note: status_code is also duplicated in HTTP status code

Example using curl:

curl -X POST http://127.0.0.1:55555/api/v1/ -H "Content-Type: application/json" --data '{"model": "proxy", "method": "get"}'

Example using httpie:

http POST http://127.0.0.1:55555/api/v1/ model=proxy method=get

Example using python's requests library:

import requests
import json


def get_proxies():
    result = []
    json_data = {
        "model": "proxy",
        "method": "get",
    }

    response = requests.post('http://127.0.0.1:55555/api/v1/', json=json_data)
    if response.status_code == 200:
        response = json.loads(response.text)
        for proxy in response['data']:
            result.append(proxy['address'])
    else:
        # check error here
        pass

    return result

Example using aiohttp library:

import aiohttp


async def get_proxies():
    result = []
    json_data = {
        "model": "proxy",
        "method": "get",
    }

    async with aiohttp.ClientSession() as session:
        async with session.post('http://127.0.0.1:55555/api/v1/', json=json_data) as response:
            if response.status == 200:
                response = json.loads(await response.text())
                for proxy in response['data']:
                    result.append(proxy['address'])
            else:
                # check error here
                pass

    return result

How to interact with API?

Read more about API here

How to contribute?

TODO: write guide about it

How to test it?

If you made the changes to code and want to check that you didn't break anything, go here

How to deploy on production using supervisor, nginx and postgresql in 8 steps?

1 Install supervisor, nginx and postgresql

root@server:~$ apt install supervisor nginx postgresql

2 Create virtual environment and install requirements on it

3 Copy settings.py example:

proxy_py@server:~/proxy_py$ cp config_examples/settings.py proxy_py/

4 create unprivileged user in postgresql database and change database authentication data in settings.py

proxy_py@server:~/proxy_py$ vim proxy_py/settings.py
DATABASE_CONNECTION_KWARGS = {
    'database': 'YOUR_POSTGRES_DATABASE',
    'user': 'YOUR_POSTGRES_USER',
    'password': 'YOUR_POSTGRES_PASSWORD',
    # number of simultaneous connections
    # 'max_connections': 20,
}

5 Copy supervisor config example and change it for your case

root@server:~$ cp /home/proxy_py/proxy_py/config_examples/proxy_py.supervisor.conf /etc/supervisor/conf.d/proxy_py.conf
root@server:~$ vim /etc/supervisor/conf.d/proxy_py.conf

6 Copy nginx config example, enable it and change if you need

root@server:~$ cp /home/proxy_py/proxy_py/config_examples/proxy_py.nginx.conf /etc/nginx/sites-available/proxy_py
root@server:~$ ln -s /etc/nginx/sites-available/proxy_py /etc/nginx/sites-enabled/
root@server:~$ vim /etc/nginx/sites-available/proxy_py

7 Restart supervisor and Nginx

root@server:~$ supervisorctl reread
root@server:~$ supervisorctl update
root@server:~$ /etc/init.d/nginx configtest
root@server:~$ /etc/init.d/nginx restart

8 Enjoy using it on your server!

What is it depend on?

See requirements.txt

GitHub Stars

142

LAST COMMIT

1yr ago

MAINTAINERS

1

CONTRIBUTORS

5

OPEN ISSUES

10

OPEN PRs

2
VersionTagPublished
2.0
4yrs ago
No alternatives found
No tutorials found
Add a tutorial