SEA: Decouple Link Fetching

[varun-edachali-dbx]

What type of PR is this?

• Feature

Description

For efficiency, we fetch the links for data in the SeaCloudFetchQueue in a separate thread. Link fetches are slow because they come from the bottlenecked control plane, so coupling the fetching logic with the main process slows down our fetch phase. Instead, we proactively fetch links in the background and use them when required.

Note: this PR does not account for link expiry. This will be handled separately in #635.

How is this tested?

• Unit tests
• E2E Tests
• Manually
• N/A

Related Tickets & Documents

N/A

[github-actions]

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).