Server Structure

You'll notice that this project has several moving parts, some for scaling and some for making analysis easier. At minimum, the structure is as follows:

• A server that distributes "work"
• A client that queries "work" from the WG API
• A PostgreSQL database/server

Note that these can all be contained within one machine with three different service accounts for proper access separation. In fact, my own personal server at home does this without any issue!

Operationally, the flow works as follows:

1. The server kicks off the project and checks a local configuration file
2. The server connects to the database and creates a new "total_battles" and "diff_battles" table. If the "player" table does not exist, it is created. Two functions are created as "triggers," or actions that are executed when certain conditions are met
3. The server reads the minimum and maximum player ID numbers for Xbox and Playstation that are to be queried.
4. The server reads local files and generates a checksum for each one to compare against client copies for integrity and update delivery.
5. The client connects and requests updates from the server using the update.py script, downloading any new/missing files and removing unnecessary ones.
6. The client registers itself as ready for work with the server.
7. The server will send an updated configuration to the client, containing an API key to use, any encrypted connection instructions, and the limit on the number of queries per second that can be made without being throttled by the API.
8. The client then switches to the client.py script.
9. The client opens a websocket connection with the server for work communication.
10. The server sends work batches to the client for it to query from the WG API.
11. The client takes the starting and ending account ID number to query and fills in the missing numbers, creating a list and sending it to the API to fetch relevant data.
12. The client parses the response and prepares it for transport back to the server.
13. The client sends the data back to the server, serialized, so the server can reconstruct it and push it into a queue.
14. The server sends new work to the client.
15. Once all work is exhausted, clients are then disconnected.
16. The server will wait for all results in the queue to be passed to the database by helpers. These helpers were running in parallel processes while the server was running.
17. Once all work is pushed to the database, the server then checks if it is to dynamically grow the maximum player IDs to query. If yes, the server checks the maximum player ID in the database and compares it to the maximum ID in the configuration file. If the database max ID is within 50,000 of the config max ID, the config max ID is increased by 200,000 for that platform and written back to the file. This prevents the need of a user to check in daily and determine if the ID range needs to be increased.
18. The config is checked for ElasticSearch settings. If they exist, the server pulls changes from the database and pushes it to the listed ElasticSearch servers.

Note that while the code only supports one server, it can have any number of clients querying data. The code is written to be asynchronous with the goal of as little blocking code as possible. Clients can join at any time while there is still work to be done, and the server will preemptively send work if it is known ahead of time that the server will be heavily burdened.