Use Mod Search Database #852

otobot1 · 2024-08-24T23:49:26Z

Closes #606.

-updates the local on-disk copy of the mod search database -only works in dev -no authentication

-untested -required info is provided during the existing mod api routes -the useDownloadUrl and useModImageUrls hooks use the mod search database info if available, otherwise they use the existing logic to fetch the info from GameBanana

various other minor tweaks

something about pino.transport not being a function

couldn't find 'fs/promises'. it's a server-only module.

…chDatabase add console logs

avoid confusion with nextjs app router

making progress. builds finish now, but the `mod.getAll` tRPC endpoint is taking WAY too long and is causing issues. potentially including the build issues.

only store Maps, and only store the parameters that we use.

- batch the mod prefetches - store a highly trimmed version of the mod search database instead of the full file

contains nextjs bundle analysis

5bdf9bb didn't quite fix the webpack error apparently

-ensure that the `logs` and `cache` directories exist -ensure outdated mod search database isn't used for builds

otobot1 · 2024-08-25T00:02:10Z

@ShouvikGhosh2048 Please review when you have a chance. #766 is higher-priority, but do this whenever.
Please check that both production and dev mode work.

…axios-1.7.5 Bump axios from 1.7.2 to 1.7.5

.env.example

scripts/helperFunctions/validateAllDirectories/validateDirectory.ts

scripts/helperFunctions/validateAllDirectories/validateLogDirectory.ts

scripts/helperFunctions/validateAllDirectories/validateCacheDirectory.ts

ShouvikGhosh2048 · 2024-08-26T05:58:50Z

src/hooks/globalContexts/modDownloadUrl/useModDownloadUrl.ts

+
+    const cachedDownloadUrl = contextOrUndefined?.state[gamebananaModId];
+
+    const [downloadUrl, setDownloadUrl] = useState<string>(cachedDownloadUrl ?? "");


Do we need this state? We update the context whenever we update the downloadUrl, so couldn't we just use the cachedDownloadUrl?

Yeah I think we can. I'll take a stab at it once I fix the carousel

ShouvikGhosh2048 · 2024-08-26T06:12:51Z

src/hooks/globalContexts/modImageUrls/useModImageUrls.ts

 };


 export const useModImageUrls = (
    {
        gamebananaModId,
+        screenshotsFromModSearchDatabase,
    }: useModImageUrlsProps,
 ): ModImageUrls => {
    const contextOrUndefined = useContext(modImageUrlsContext);


Same thing here - we update state and context together, so couldn't we use the context directly and remove the imageUrls state?

src/server/api/routers/map_mod_publisher/mod.ts

src/server/gamebananaMirror/yamlHandlers/utils/getFileSystemPath.ts

src/server/gamebananaMirror/yamlHandlers/modSearchDatabase/modSearchDatabase.ts

src/server/gamebananaMirror/yamlHandlers/utils/getYamlFile.ts

ShouvikGhosh2048 · 2024-08-26T16:37:43Z

I'm getting a Failed to write the Mod Search Database to the file system. when I refresh the mods page (in dev).
I think the GAMEBANANA_MOD_IMAGES_BASE_URL should be "https://images.gamebanana.com/img/ss/mods/"?

otobot1 · 2024-08-26T19:22:09Z

I'm getting a Failed to write the Mod Search Database to the file system. when I refresh the mods page (in dev).

I think the GAMEBANANA_MOD_IMAGES_BASE_URL should be "https://images.gamebanana.com/img/ss/mods/"?

I bet that warning is because you don't have the local copy of the mod search database. If it's the warning I think it is, it will just download the latest copy and carry on. I just updated the warning so it's more clear that it's ok. Please let me know if it was a different error.
That was the original value for the images base url actually, but I had to change it to be more general as a couple of mods in the mod search database have their images elsewhere at images.gamebanana.com. We consume the whole url from the mod search database and just use this constant as a sanity check for the beginning of the url, so it's not a big deal.

otobot1 · 2024-08-27T03:19:17Z

Looks like I broke the carousel at some point on this branch (the arrows don't do anything now). I'll fix that and then take a stab at those remaining comments.

otobot1 added 20 commits August 10, 2024 14:26

fix auth error

76e45cf

remove default paths for Maddie's JSONs

c315afb

update everest update database path env variable name

bfe3bad

add mod-search-database dev api endpoint

9e98892

-updates the local on-disk copy of the mod search database -only works in dev -no authentication

adjust GAMEBANANA_MOD_IMAGES_BASE_URL to be broader

c6d48b3

various other minor tweaks

split code to improve tree shaking and resolve webpack error

5bdf9bb

something about pino.transport not being a function

resolve module not found next build error

4b2c2bd

couldn't find 'fs/promises'. it's a server-only module.

fix mod search database not validating when calling getCurrentModSear…

09f6925

…chDatabase add console logs

resolve largest contentful paint warning from nextjs

572db42

rename appRouter to apiRouter

3b7253b

avoid confusion with nextjs app router

use path module to avoid file system errors

6555204

continue troubleshooting builds

be362ea

making progress. builds finish now, but the `mod.getAll` tRPC endpoint is taking WAY too long and is causing issues. potentially including the build issues.

reduce size of locally cached mod search database

f03cd09

only store Maps, and only store the parameters that we use.

increase performance of the mods page

62244b0

- batch the mod prefetches - store a highly trimmed version of the mod search database instead of the full file

ignore /analyze

f458de2

contains nextjs bundle analysis

fetch mod search db if it isnt already present instead of erroring

518599d

split code to improve tree shaking

63c48f0

5bdf9bb didn't quite fix the webpack error apparently

make tsx a regular (non-dev) dependency

d6e5694

add predev and prebuild scripts

24e5060

-ensure that the `logs` and `cache` directories exist -ensure outdated mod search database isn't used for builds

otobot1 linked an issue Aug 24, 2024 that may be closed by this pull request

Use Maddie's Random Stuff #606

Open

Merge branch 'main' into use-mod-search-database

7fb6c65

otobot1 requested a review from ShouvikGhosh2048 August 25, 2024 00:00

Merge pull request #853 from celestemods-com/dependabot/npm_and_yarn/…

dfaa8b4

…axios-1.7.5 Bump axios from 1.7.2 to 1.7.5