Improve compile time on dev for huge vaults/content folders

[olayway]

From Next.js docs:

If a page has Dynamic Routes and uses getStaticProps, it needs to define a list of paths to be statically generated.
When you export a function called getStaticPaths (Static Site Generation) from a page that uses dynamic routes, Next.js will statically pre-render all the paths specified by getStaticPaths.

In Flowershow, in order to support any user's content folder (vault) structure, we only have a single catch-all [[...slug]].jsx page, which creates a list of paths to the user's files in its getStaticPaths by taking ALL the documents generated by Contentlayer and using their URL paths.

pages
├── _all.jsx
├── _app.jsx
└── [[...slug]].jsx

Because of that, the more files in the user's vault the slower the initial compilation time, as Next.js will pre-render all the pages returned by getStaticPaths.

If the /pages folder reflected the content folder structure, with each folder having its own [[...slug]].jsx file, this time would be split, i.e. when requesting localhost:3000/ only the files in the root folder would be pre-rendered, then, when requesting e.g. localhost:3000/docs/some-tutorial-page, only the files inside docs folder would be pre-rendered etc. The issue would persist though if the user had a flat folder structure, i.e. there were no subdirectories in the content folder.

This could probably be achieved with some script.

[rufuspollock]

Good point to raise. Think would be useful to get more evidence that this is an issue for people. Also could to consider multiple options for improving performance here - e.g. maybe turbo will help a lot etc.

[olayway]

I have tested it a couple of times, each time increasing the number of md files and it does seem to be affecting compilation time. Which makes sense based on how Next.js pre-renders dynamic pages.

[rufuspollock]

👍 the question is how bad is the problem both in terms of how many people is this an issue for and how bad is that problem for them. Roughly:

Importance to solve = How bad does this get for X notes in a vault x Number of people with X or more notes in the vault

[danymat]

Personally there's two things preventing me for properly using flowershow:

• Difficulties in building/previewing app (reference to other issues)
• Very slow compiling/building (this issue)

Actually, building time is not as bad as previewing time. I personally think 600 notes can be quite common nowadays, specially for people that create map of contents and structure their obsidian with more notes. On my end I don't actually use MOC, but even though every note is an idea, growing the number of notes is inevitable.

• Let's suppose that very few people have a number of notes that is difficult for them to generate: more time passes, more people will eventually get back to this problem and view it as a no go.
• Or suppose lots of joiners already have a high number of notes: then as the project gets more coverage, this question will arise very quickly.

The subsidiary question supposing these 2 points is then:
Do we want to wait for other issues to be resolved before going into this one, or should we tackle it asap ?

I'm not calling for action as I know building an app has a lot of requirements, but this issue can quickly turn into a nogo to people that already have "big numbers" (lol), and will eventually turn into a nogo to others that will increase their numbers

[rufuspollock]

@danymat on the preview - i assume this on first boot. Once the preview command is running are reloads fast? (they should be but maybe they aren't).

Also in terms of "very slow compile time" can you give a sense of what it is for you. e.g. is it 5s, 10s, 60s?

[danymat]

This is the result time of building over the repo:

➜  Developer time npx flowershow@latest build

> @flowershow/[email protected] build
> next build

...
...
info  - Generating static pages (603/603)
npx flowershow@latest build  73.66s user 4.62s system 83% cpu 1:33.98 total