feat(server): library refresh go brrr #14456

etnoy · 2024-12-02T21:37:45Z

This PR significantly improves library scanning performance. Wherever suitable, we are doing jobs in batches, and many looped database interactions are replaced with SQL queries.

User testimonials
"@etnoy what on earth have you done. I tried your PR and it finished the scan for 1M assets in 37 seconds down from 728s on main. It takes 188s just to finish queuing on main" -- @mertalev

Changes made

When crawling a library: Don't call stat() on the files when importing, instead put a placeholder date (9999-12-31) that will be overwritten by metadata extraction. All dates after year 9000 are ignored in the timeline
When importing new library files: Crawl 10k files per batch and then insert each batch in a single SQL query instead adding them individually
When checking asset filenames against import paths and exclusion patterns: Do this directly in SQL which is much faster
When a library is deleted, hide all its assets first. Otherwise, they'll linger until the asset deletion job catches up which can take days for >1M libraries

Plus several minor cleanups and performance enhancements.

The performance improvements are at least an order of magnitude in library scanning.

Benchmark 1
A library scan with 22k items where nothing has changed since the last scan used to take 1m 22s, now it's below 10 seconds, an improvement of 87 percent!

Benchmark 2
A clean library import with 19k items takes 1m40s in main and 7 seconds in this PR.
NOTE: this benchmark is only the library service scan and does not include the metadata extraction. Also, some fs calls have been migrated from the library service to the metadata service, although this should only have a minor impact on overall scan performance

Benchmark 3
Importing a library with >5M assets.

Time to 1M imported (without metadata extraction): 6m50s.

No need to compare to main, you know it's fast!

Benchmark 4
Importing a library of 527041 files took 1m58s (without metadata extraction) in this PR.
No need to compare to main, you know it's fast!

Bonus:

Greatly improved log messages related to library scans.
This scan imports all new files:

This is an "idle scan", where a refresh finds no changes:

More e2e tests for handling when offline files go back online, leading to one major bug fixed

Future work:

Crawl for XMP sidecars instead of queuing a sidecar discovery for each asset

Final note:
This PR allowed me to hit a milestone of 10M assets in a single Immich instance, likely a world-first. This does require max-old-space-size=8096, but that's to be expected anyway

mertalev

Nice start! I think there are still a lot of untapped potential improvements here.

server/src/services/library.service.ts

mertalev · 2024-12-04T18:08:32Z

The update to fileCreatedAt, fileModifiedAt and originalFileName is unnecessary and can be handled in metadata extraction since this will be queued anyway. This makes the batched update for isOffline and deletedAt simpler since there'll be no values that are unique to each asset.

etnoy · 2024-12-08T22:20:59Z

Thanks for your comments @mertalev ! I'll first attempt to do the import path and exclusion pattern checks in SQL and then move to your suggestions

server/src/repositories/asset.repository.ts

server/src/services/library.service.ts

etnoy · 2024-12-12T21:10:28Z

The update to fileCreatedAt, fileModifiedAt and originalFileName is unnecessary and can be handled in metadata extraction since this will be queued anyway. This makes the batched update for isOffline and deletedAt simpler since there'll be no values that are unique to each asset.

Never thought of that, I've implemented your suggestion. I'm also considering changing the initial import code to ignore file mtime, this allows us to not do any file system calls except for the crawl. Metadata extraction will have to do the heavy lifting instead

mertalev · 2024-12-12T21:26:49Z

The update to fileCreatedAt, fileModifiedAt and originalFileName is unnecessary and can be handled in metadata extraction since this will be queued anyway. This makes the batched update for isOffline and deletedAt simpler since there'll be no values that are unique to each asset.

Never thought of that, I've implemented your suggestion. I'm also considering changing the initial import code to ignore file mtime, this allows us to not do any file system calls except for the crawl. Metadata extraction will have to do the heavy lifting instead

Would that mean you queue them for metadata extraction even if they're unchanged? You can test it but I think it'd be more overhead than the stat calls.

Edit: also if you do this with the source set to upload, it would definitely be worse because it would queue a bunch of other things after metadata extraction.

etnoy · 2024-12-12T21:42:08Z

The update to fileCreatedAt, fileModifiedAt and originalFileName is unnecessary and can be handled in metadata extraction since this will be queued anyway. This makes the batched update for isOffline and deletedAt simpler since there'll be no values that are unique to each asset.

Never thought of that, I've implemented your suggestion. I'm also considering changing the initial import code to ignore file mtime, this allows us to not do any file system calls except for the crawl. Metadata extraction will have to do the heavy lifting instead

Would that mean you queue them for metadata extraction even if they're unchanged? You can test it but I think it'd be more overhead than the stat calls.

Edit: also if you do this with the source set to upload, it would definitely be worse because it would queue a bunch of other things after metadata extraction.

I was referring to new imports, files that are new to immich. I hoped to improve the ingest performance by removing the stat call. After testing, there are two issues:

assetRepository.create requires mtime, which we can only get from stat. We could work around that by setting it to new Date(), but ideally it should be undefined
We still check for the existence of a sidecar, and this complicates things

If we can mitigate the two issues above, I can rewrite the library import feature and do that in batches as well!

mertalev · 2024-12-12T21:55:06Z

I don't see why fileModifiedAt needs a non-null constraint in the DB. Might just be an oversight that didn't matter because it didn't affect our usage. I think you can change the asset entity and generate a migration to remove that constraint.

For sidecar files, maybe you could add.xmp to the glob filter and enable the option to make the files come in sorted order? That way you could make sure they're in the same batch.

etnoy · 2024-12-12T22:20:59Z

I don't see why fileModifiedAt needs a non-null constraint in the DB. Might just be an oversight that didn't matter because it didn't affect our usage. I think you can change the asset entity and generate a migration to remove that constraint.

For sidecar files, maybe you could add.xmp to the glob filter and enable the option to make the files come in sorted order? That way you could make sure they're in the same batch.

I might just put new Date() in at the moment to keep the PR somewhat constrained.

Regarding sidecars, I have thought about that, problem right now is that we're batching the crawled files in batches of 10k. It might be hard to do get that working alright. Maybe I'll just queue a sidecar discovery for every imported asset for now

…-app/immich into feat/inline-offline-check

danieldietzler · 2025-02-24T20:06:50Z

server/src/repositories/asset.repository.ts

+      .limit(pagination.take + 1)
+      .offset(pagination.skip ?? 0);


@etnoy didn't you want to use a stream?

danieldietzler · 2025-02-24T20:12:42Z

server/src/repositories/asset.repository.ts

+  async getLibraryAssetCount(options: AssetSearchOptions = {}): Promise<number | undefined> {
+    const { count } = await this.db
+      .selectFrom('assets')
+      .select(sql`COUNT(*)`.as('count'))


Still an open comment.

danieldietzler · 2025-02-24T20:16:40Z

server/src/services/library.service.ts

+    const assetIds: string[] = [];
+
+    for (let i = 0; i < assetImports.length; i += 5000) {
+      // Chunk the imports to avoid the postgres limit of max parameters at once


Is this still a problem or does kysely already handle this?

I think the same limitation applies to something like this.db.insertInto('assets').values(assets). But it does let us pass the entire array as a single parameter (meaning the 65,535 limit doesn't apply) if it's written differently.

@danieldietzler when I first migrated this PR to kysely I had to batch it here to avoid errors

server/src/services/library.service.ts

danieldietzler · 2025-02-24T20:24:56Z

server/src/services/library.service.ts

-      return JobStatus.SKIPPED;
-    }
+  @OnJob({ name: JobName.LIBRARY_SYNC_ASSETS, queue: QueueName.LIBRARY })
+  async handleSyncAssets(job: JobOf<JobName.LIBRARY_SYNC_ASSETS>): Promise<JobStatus> {


In this function (and kind of in general) there is a lot of logic (or at least lines of code I need to read) that are purely added complexity for the sake of logs. I get that logs are neat, but I would personally argue that some of them don't add any value at all. IMO logs should primarily indicate and help understand an error.

Trust me, if you scan or rescan a library with >1M assets you really need these logs

Can we at least reduce them or have more generic logs instead of dedicated if/else structures and variables just for logging?

Done. Let me know if you think this is better

server/src/services/library.service.ts

…/inline-offline-check

alextran1502 · 2025-03-03T03:31:10Z

You call on merging this one @mertalev

etnoy added the changelog:enhancement label Dec 2, 2024

github-actions bot added the 🗄️server label Dec 2, 2024

etnoy force-pushed the feat/inline-offline-check branch from 0eb1440 to 80aa615 Compare December 2, 2024 21:45

feat: run all offline checks in a single job

8ecde3b

etnoy force-pushed the feat/inline-offline-check branch from 80aa615 to 8ecde3b Compare December 2, 2024 21:46

mertalev reviewed Dec 4, 2024

View reviewed changes

server/src/services/library.service.ts Outdated Show resolved Hide resolved

server/src/services/library.service.ts Outdated Show resolved Hide resolved

etnoy force-pushed the feat/inline-offline-check branch 2 times, most recently from d394654 to 8b2a48c Compare December 9, 2024 21:34

Merge remote-tracking branch 'origin' into feat/inline-offline-check

02c5765

etnoy force-pushed the feat/inline-offline-check branch 3 times, most recently from 6d69307 to c26f6aa Compare December 10, 2024 16:41

etnoy added the changelog:bugfix label Dec 10, 2024

etnoy force-pushed the feat/inline-offline-check branch from c26f6aa to a3be620 Compare December 10, 2024 20:39

etnoy removed the changelog:bugfix label Dec 10, 2024

etnoy changed the title ~~feat(server): run all offline checks in a single job~~ feat(server): library refresh go brrr Dec 10, 2024

mertalev reviewed Dec 10, 2024

View reviewed changes

server/src/repositories/asset.repository.ts Outdated Show resolved Hide resolved

server/src/repositories/asset.repository.ts Show resolved Hide resolved

mertalev reviewed Dec 10, 2024

View reviewed changes

server/src/services/library.service.ts Outdated Show resolved Hide resolved

etnoy force-pushed the feat/inline-offline-check branch 5 times, most recently from 775b817 to 69b273d Compare December 12, 2024 20:59

etnoy force-pushed the feat/inline-offline-check branch 5 times, most recently from 99c8e91 to d00913b Compare February 17, 2025 22:36

Merge branch 'feat/inline-offline-check' of https://github.com/immich…

462dafa

…-app/immich into feat/inline-offline-check

etnoy force-pushed the feat/inline-offline-check branch 2 times, most recently from 250b04c to 53324da Compare February 17, 2025 23:51

Merge remote-tracking branch 'origin' into feat/inline-offline-check

af20495

etnoy force-pushed the feat/inline-offline-check branch 2 times, most recently from 447fb26 to bc1c22d Compare February 18, 2025 00:15

Merge remote-tracking branch 'origin' into feat/inline-offline-check

f35866a

etnoy force-pushed the feat/inline-offline-check branch from bc1c22d to f35866a Compare February 18, 2025 00:20

etnoy requested a review from jrasm91 February 22, 2025 23:20

danieldietzler reviewed Feb 24, 2025

View reviewed changes

etnoy force-pushed the feat/inline-offline-check branch 7 times, most recently from 23c3994 to 17bd7ec Compare February 27, 2025 00:51

Merge remote-tracking branch 'origin' into feat/inline-offline-check

eb3a705

etnoy force-pushed the feat/inline-offline-check branch from 17bd7ec to cb772ad Compare February 27, 2025 09:02

Merge branch 'main' of https://github.com/immich-app/immich into feat…

aa689ef

…/inline-offline-check

etnoy force-pushed the feat/inline-offline-check branch from cb772ad to aa689ef Compare February 27, 2025 10:49

Merge branch 'main' of https://github.com/immich-app/immich into feat…

d8d61a0

…/inline-offline-check

etnoy force-pushed the feat/inline-offline-check branch from 9313217 to d8d61a0 Compare February 28, 2025 12:53

Merge remote-tracking branch 'origin' into feat/inline-offline-check

5968d63

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(server): library refresh go brrr #14456

feat(server): library refresh go brrr #14456

etnoy commented Dec 2, 2024 •

edited

Loading

mertalev left a comment

mertalev commented Dec 4, 2024

etnoy commented Dec 8, 2024

etnoy commented Dec 12, 2024

mertalev commented Dec 12, 2024 •

edited

Loading

etnoy commented Dec 12, 2024

mertalev commented Dec 12, 2024 •

edited

Loading

etnoy commented Dec 12, 2024

danieldietzler Feb 24, 2025

danieldietzler Feb 24, 2025

danieldietzler Feb 24, 2025

mertalev Feb 24, 2025

etnoy Feb 27, 2025

danieldietzler Feb 24, 2025

etnoy Feb 24, 2025

danieldietzler Feb 25, 2025

etnoy Feb 26, 2025

alextran1502 commented Mar 3, 2025

feat(server): library refresh go brrr #14456

Are you sure you want to change the base?

feat(server): library refresh go brrr #14456

Conversation

etnoy commented Dec 2, 2024 • edited Loading

mertalev left a comment

Choose a reason for hiding this comment

mertalev commented Dec 4, 2024

etnoy commented Dec 8, 2024

etnoy commented Dec 12, 2024

mertalev commented Dec 12, 2024 • edited Loading

etnoy commented Dec 12, 2024

mertalev commented Dec 12, 2024 • edited Loading

etnoy commented Dec 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alextran1502 commented Mar 3, 2025

etnoy commented Dec 2, 2024 •

edited

Loading

mertalev commented Dec 12, 2024 •

edited

Loading

mertalev commented Dec 12, 2024 •

edited

Loading