Skip to content

Commit d4fca11

Browse files
committed
update structure
1 parent 39bcf6b commit d4fca11

29 files changed

Lines changed: 429 additions & 400 deletions

.gitignore

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,5 @@
22
*.pyc
33
*.db
44
venv
5-
/.idea
5+
/.idea
6+
*.log

LICENSE

Lines changed: 0 additions & 201 deletions
This file was deleted.

README.md

Lines changed: 52 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,52 +1,76 @@
11
# ProxyPool
22

3-
## 安装
3+
## Requirements
44

5-
### 安装Python
5+
* Docker
66

7-
至少Python3.5以上
7+
or
88

9-
### 安装Redis
9+
* Python: >=3.6
10+
* Redis
11+
* Environment: Virtual Env
1012

11-
安装好之后将Redis服务开启
13+
## Run with Docker
1214

13-
### 配置代理池
14-
15-
```
16-
cd proxypool
15+
```shell script
16+
docker-compose up
1717
```
1818

19-
进入proxypool目录,修改settings.py文件
19+
## Run without Docker
2020

21-
PASSWORD为Redis密码,如果为空,则设置为None
21+
Here are steps to run ProxyPool.
2222

23-
#### 安装依赖
23+
### Install Redis
2424

25-
```
26-
pip3 install -r requirements.txt
25+
You need to install Redis locally or get a Redis server firstly.
26+
27+
Next set Redis environment:
28+
29+
```shell script
30+
export REDIS_HOST='localhost'
31+
export REDIS_PORT=6379
32+
export REDIS_PASSWORD='foobar'
2733
```
2834

29-
#### 打开代理池和API
35+
Also you can just set the Redis Connection String:
3036

37+
```shell script
38+
export REDIS_CONNECTION_STRING='redis://[password]@host:port'
3139
```
32-
python3 run.py
40+
41+
You can choose one method of above to set Redis environment.
42+
43+
### Clone ProxyPool
44+
45+
```shell script
46+
https://github.com/Python3WebSpider/ProxyPool
47+
cd ProxyPool
3348
```
3449

35-
## 获取代理
50+
### Install Requirements
51+
52+
```shell script
53+
pip3 install -r requirements.txt
54+
```
3655

56+
### Run ProxyPool
3757

38-
利用requests获取方法如下
58+
You can run all of the processors including Getter、Tester、
59+
Server:
3960

40-
```python
41-
import requests
61+
```shell script
62+
python3 run.py
63+
```
4264

43-
PROXY_POOL_URL = 'http://localhost:5555/random'
65+
or run with args to run specific processor:
4466

45-
def get_proxy():
46-
try:
47-
response = requests.get(PROXY_POOL_URL)
48-
if response.status_code == 200:
49-
return response.text
50-
except ConnectionError:
51-
return None
67+
```shell script
68+
python3 run.py --processor getter
69+
python3 run.py --processor tester
70+
python3 run.py --processor server
5271
```
72+
73+
### Usage
74+
75+
After running the ProxyPool, you can visit
76+
[http://localhost:5555/random](http://localhost:5555/) to access random proxy.

importer.py

Lines changed: 0 additions & 4 deletions
This file was deleted.

proxy provider.txt

Lines changed: 0 additions & 7 deletions
This file was deleted.

0 commit comments

Comments
 (0)