Repository navigation
Conversation
`Store::check` opens the index and reads and decodes every covering filter on each call, which is nearly all of its ~7.5 ms. `Store::load` reads the index and all its filters once; `LoadedStore::check` then answers from memory in a few microseconds, and `is_stale` tells a long-running process when a fetch has replaced the index. Both paths run the same traversal: `Index` now abstracts where its entry sections come from (seeking the file, or the file held in memory) and where filters come from, so verdicts cannot drift between them. Loading tolerates a concurrent fetch per the C2SP reader rule: a filter the index names but the disk lacks means the index was replaced, so the load starts over with the new one.
Merging this PR will not alter performance1 warning
Performance Changes
Comparing |
|
I think it makes sense to do something like this. In my mind, this would happen internal to the If we're going to review this, I think it needs to be broken up into some smaller commits. |
|
Thanks for taking a look!
Makes sense, will change!
Not sure on this one - reloading requires a FS check (either to load new content or to do mtime check, etc) which I don't think the call to check should be paying. In my mind this is something that should be left up to a consumer to decide on when it wants to check freshness and reload.
I can split. I assume 1 commit = 1 pr? |
In order to be misuse-resistant, I think we should probably at least check freshness and reflect it in the return type (adding a
I think we'd prefer multiple commits working towards a single goal being in a single PR together (but no fixup commits etc, see |
Store::checkopens the index and reads and decodes every covering filter on each call, which is nearly all of its ~7.5 ms.Store::loadreads the index and all its filters once;LoadedStore::checkthen answers from memory in a few microseconds, andis_staletells a long-running process when a fetch has replaced the index.Both paths run the same traversal:
Indexnow abstracts where its entry sections come from (seeking the file, or the file held in memory) and where filters come from, so verdicts cannot drift between them.Loading tolerates a concurrent fetch per the C2SP reader rule: a filter the index names but the disk lacks means the index was replaced, so the load starts over with the new one.