Configurable caching?

[huw0]

In #11 and CLAUDE.md in-memory caching is mentioned.

Does metadata need to be more configurable so that this can be refreshed whenever git is synced to storage without a Trino restart?

I'd propose that metadata refresh could be strictly time based. However when doing a SELECT, if the view isn't in the cache, the connector would still attempt to look it up from storage? This ensures that queries will continue to work even when the cache is stale.

The standard Trino JDBC connectors work in this way by allowing 'hidden' tables to be retrieved that aren't present in the metadata.

[robfromboulder]

Another great suggestion, thanks!

Having to restart Trino is definitely a pain point

I agree that there are a couple of options that could have value:

• Refreshing the cache periodically on a timer
• Re-checking the filesystem contents if a cache miss is detected
• Exposing a scalar function that syncs to the filesystem on demand

Stupid question for you just to calibrate -- is this more of a testing use case? Asking because for testing it might be appropriate to be more deliberate (ie scalar function) but for runtime/production I can see this needing to be done more automatically (ie timer).

[huw0]

Stupid question for you just to calibrate -- is this more of a testing use case?

I definitely see this as being a production use-case. For example, assuming the git-sync scenario for views and a Kubernetes environment...

• Trino is deployed using the official Helm chart.
• Viewzoo sync from git occurs using a Kubernetes cronjob.

By having Trino refresh the views on a timer, this simplifies the process.

Exposing a scalar function that syncs to the filesystem on demand

I had not considered this option but actually I can see this being helpful in both a test or production scenario. As the sync would be able to call this function for an immediate refresh.

Another option might be to expose a custom endpoint under the system API - however I think the scalar function is a better fit to Trino and allows for more granular access control.