Filter other Pangea events from sync

[wcjord]

What other Pangea events can we filter from sync and fetch only when they are needed?

A prime candidate is the results of our text to speech. These are big events and only needed when a user goes to access the read aloud functionality.

Another would be the representations made for translations to an L1.

Certainly others though those may be the most numerous/large.

[ggurdin]

Initial sync with pangea.representation events filtered out took 1 second longer than a sync without that filtering.

Tested filtering out pangea.representations, which are by far the most numerous of these, but with the function to update analytics already manually loading for every chat that the logged in user is in (these manual calls can't be filtered with the current SDK setup), removing them from the syncs doesn't seem to make a difference in how many are loaded.

[wcjord]

@clokep This is rather surprising. Why would syncing with the server take more time if it's filtering some events? Is it loading all events into memory from the database then cycling through them? Is this something we could hope to optimize if we were to make changes to synapse?

[clokep]

Why would syncing with the server take more time if it's filtering some events? Is it loading all events into memory from the database then cycling through them? Is this something we could hope to optimize if we were to make changes to synapse?

It should be filtering them at the database level. I'm unsure why they would be taking longer, but we can take a look at the logs and/or query plans of the database if you have two example requests. The only thing that maybe could affect this is if filtering skips a cache somewhere, but that sounds like a bug.

Naively I would definitely expect that returning less data == a faster response.

[wcjord]

@clokep ok that is confusing. we may want you to look further into and will provide some example requests in that case. for now, we're working on some other optimizations that seem to potentially offer more gains.