Caching and UPDATED-sorting for the Github Data Sync Algorithm

[GODrums]

Motivation

The current data fetching algorithm didn't collect all the required data and had massive performance issues. This PR aims to fix both of these issues. With this PR, the table should be on-par with the data from the original Python script.

Description

This PR includes multiple improvements:

• Github PR data is now sorted after UPDATED instead of CREATED, allowing us to get all PR that were worked on
• Instead of always querying for fetched entities in the DB, we implement caching for improved performance (see performance test below)
• GET /leaderboard now filters the timeframe for the data before calculating the response

Performance Test

#PRs	develop-branch	PR (no caching)	PR (caching)
8	90s	-	-
53	impossible	5 min	3 min

In my test runs the new fetching process now takes about 3 minutes (53 PRs), in comparison to the original 1 minute (8 PRs) and 5 minutes without caching (53 PRs).

Screenshots (if applicable)

Checklist

General

• PR title is clear and descriptive
• PR description explains the purpose and changes
• Code follows project coding standards
• Self-review of the code has been done
• Changes have been tested locally

Server (if applicable)

• Code is performant and follows best practices
• No security vulnerabilities introduced
• Proper error handling has been implemented
• Added tests for new functionality
• Changes have been tested in different environments (if applicable)

[FelixTJDietrich]

Code looks good, except idk what happens with dismissed reviews. Only thing that is missing is the fix in the story so the check runs through

[FelixTJDietrich]

What happens with the dismissed reviews?

[GODrums]

Interesting point! They were counted as comments for now. This should be fixed in: #97.

In the future we might want to do some sort of state replacement similar to the original Python script.

[FelixTJDietrich]

I think everything is fine for now, in the future we might want store the score for each activity also in the database and then we can just calculate rank and aggregated score more easily.

For example, a pull request review comes in via webhooks, we estimate the effort it took to do this review and store it in the database. I think as it is now the score is not stable since the PR might get more changes and reviews get dismissed.

[GODrums]

Yes, this is very true.
We for sure have to improve a few elements for a correct and performant long-running execution.