-
Notifications
You must be signed in to change notification settings - Fork 17
/
FAQ
58 lines (49 loc) · 2.73 KB
/
FAQ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
What is the directory structure?
/
+- bin/
| +- services/
| | +- Services, it is possible to start them by hand but the initscripts do normally the job
| +- Some scripts/wrappers to generate some particular queries
+- doc/
| +- UML Diagrams, dump of the wiki, documentation on third party softwares, blog articles...
+- etc/
| +- init.d/
| | +- Initscripts
| +- bgp-ranking.conf config file used by the initscripts and define all the paths of the application and some constraints
| +- redis.conf config file of redis
+- lib/
| +- db_init/
| | +- Scripts to initialize the databases (whois client and ranking)
| +- db_models/
| | +- Databases models (whois client and ranking)
| +- helpers/
| | +- some helpers (for the initscripts and the manipulation of the ips) used very often in the project
| +- modules/
| | +- all the modules to parse the different raw files
| +- whois_client/
| | +- whois fetcher and connector for the whois client
| +- whois_parser/
| | +- parsers to extract particular informations of the whois response
| +- fetch_RIS.py : push the RIS whois responses in the database
| +- fetch_Whois.py : push the whois responses in the database
| +- input_reader.py : pop the new entries from redis and push it in MySQL
+- scripts/ : bunch of scripts used to start the system
+- var/
| +- raw_data/
| | +- source_name/ : name of the source of the file
| | | +- old/ : directory to put the old files (already parsed)
| | | +- temp/ : directory where the lists are downloaded
| | | +- file(s) to parse
| +- run/
| | +- pid files of the running processes
| +- log/
| | +- log files
Why do we need many redis servers ? Is it not overkill ? Redis can handle a lot of query/sec.
tl;dr: if you do not use the daily dataset of dshield, one redis server is enough. Set all the redis ports to 6379 in the config file.
And comment the line with DshieldDaily!
Detailed answer:
It depends. If you use the whole system with all the sources, not. At all. Or you will see redis die.
The system is massively multiprocessed, with massively I mean around 40/50 processes when it is busy. Each of this processes interact more or less with redis. Some of them write stuff but the most "redis-intensive" work is reading information from the databases.
That is why there is two slave databases which only handle this part.
The master server handle all the write processes.
The second master redis server contains the temporary (RIS) Whois entries, it is quite intensive but a single server doing only this is enough. It is great to have this second server because many information are added/removed from the server and the other master server was quite busy to replicate the entries on the slaves.