feat(server): library refresh go brrr #14456
Conversation
etnoy
force-pushed
the
feat/inline-offline-check
branch
from
0eb1440 to
80aa615
Compare
etnoy
force-pushed
the
feat/inline-offline-check
branch
from
80aa615 to
8ecde3b
Compare
| } | ||
|
|
||
| private async handleSyncAsset(id: string, importPaths: string[], exclusionPatterns: string[]): Promise<JobStatus> { | ||
| const asset = await this.assetRepository.getById(id); | ||
| if (!asset) { | ||
| return JobStatus.SKIPPED; | ||
| } | ||
|
|
||
| const markOffline = async (explanation: string) => { |
There was a problem hiding this comment.
Just log directly without this function since the offline status will be set at the batch level at the end.
| @OnJob({ name: JobName.LIBRARY_SYNC_ASSETS, queue: QueueName.LIBRARY }) | ||
| async handleSyncAssets(job: JobOf<JobName.LIBRARY_SYNC_ASSETS>): Promise<JobStatus> { | ||
| for (const id of job.ids) { | ||
| await this.handleSyncAsset(id, job.importPaths, job.exclusionPatterns); |
There was a problem hiding this comment.
Fetch all the assets first with getByIds. Group the ones to be marked offline, queued for metadata extraction, etc. as you check them, then do batched async calls at the end as needed. The try/catch for stat should still be scoped to the asset so one error won't torpedo the batch.
|
The update to |
|
Thanks for your comments @mertalev ! I'll first attempt to do the import path and exclusion pattern checks in SQL and then move to your suggestions |
etnoy
force-pushed
the
feat/inline-offline-check
branch
2 times, most recently
from
d394654 to
8b2a48c
Compare
etnoy
force-pushed
the
feat/inline-offline-check
branch
3 times, most recently
from
6d69307 to
c26f6aa
Compare
etnoy
force-pushed
the
feat/inline-offline-check
branch
from
c26f6aa to
a3be620
Compare
etnoy
changed the title
| .where({ isOffline: false }) | ||
| .andWhere( | ||
| new Brackets((qb) => { | ||
| qb.where('originalPath NOT SIMILAR TO :paths', { |
There was a problem hiding this comment.
Use LIKE instead of SIMILAR TO.
The exclusions and import paths are also specific to a particular library, right? So you need to specify the library in the query.
Also, can you generate SQL for this and confirm with EXPLAIN ANALYZE that it uses an index?
| .update() | ||
| .set({ | ||
| isOffline: true, | ||
| deletedAt: new Date(), |
There was a problem hiding this comment.
The status also needs to be set. This is why I don't really like the status field. The same info is stored in multiple places so it's so easy for it to go out of sync like this.
| @OnJob({ name: JobName.LIBRARY_SYNC_ASSETS, queue: QueueName.LIBRARY }) | ||
| async handleSyncAssets(job: JobOf<JobName.LIBRARY_SYNC_ASSETS>): Promise<JobStatus> { | ||
| for (const id of job.ids) { | ||
| const asset = await this.assetRepository.getById(id); |
etnoy
force-pushed
the
feat/inline-offline-check
branch
from
a3be620 to
ef4db4e
Compare
For a library we currently queue one job per library asset to check if it is still online. It is more efficient to create one job per 10k assets instead, making it a tighter loop inside the library service than to create 10k tiny jobs
Other things changed:
A quick performance test for an external library with 22573 assets on my 32-core server got the following stats for a rescan
This performance improvement is important since libraries rarely change and a library rescan is run quite often.