Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bookmark polling strategy for the server #43

Open
xuatz opened this issue May 2, 2023 · 0 comments
Open

bookmark polling strategy for the server #43

xuatz opened this issue May 2, 2023 · 0 comments
Labels
Milestone

Comments

@xuatz
Copy link
Owner

xuatz commented May 2, 2023

Depends on #39

In xbo, we store a cache of the bookmarks from the original providers for a few reasons:

  • users of xbo may have a ridiculous amount of bookmarks, and providers' api may not be able to serve us fast enough cheap enough
  • we also would like to do additional processing on top of these bookmarks in the background when the user is not active (e.g. generate report, curate a sublist, group bookmarks from different providers)
  • a naive approach when trying to request for bookmarks from providers cause us to exceed the api call limit and get us rate limited or banned

So when a user has providers connected, xbo should on a schedule make a request to respectively each provider, to query to see if there are:

  • new bookmarks
  • updated/missing bookmarks
    • missing bookmarks because the users might edit/move/delete bookmarks on the provider's own website
      we need to consider how we will handle this, perhaps ignore, flag it or something smarter

And depending on the API, they way they handle cursor/pagination and updated since may be different, so we will need to implement a strategy for each of them.

And then ultimately, store these bookmarks in pocketbase and make it available for the UI to query and render.

At this point of time, I think that the bookmarks should not be transformed, but kept and stored as-is from the providers. Then we can have a 1-1 record that handles the metadata of said bookmark, for example if it is archived, soft-deleted, hard-deleted, category, etc.

@xuatz xuatz added this to the v3 milestone May 2, 2023
@xuatz xuatz added the backend label May 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant