support/datastore: add read-only http datastore.

[overcat]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've reviewed the changes in this PR and if I consider them worthwhile for being mentioned on release notes then I have updated the relevant CHANGELOG.md within the component folder structure. For example, if I changed horizon, then I updated (services/horizon/CHANGELOG.md. I add a new line item describing the change and reference to this PR. If I don't update a CHANGELOG, I acknowledge this PR's change may not be mentioned in future release notes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

What

Support HTTP as a read-only data source

Why

See #5808

Known limitations

N/A

[Copilot]

Pull Request Overview

This PR adds HTTP as a read-only data source to the datastore package, allowing access to files hosted on HTTP(S) endpoints. The implementation supports authentication via custom headers and provides all standard read operations while explicitly rejecting write operations.

• Adds HTTPDataStore implementation with configurable base URL, timeout, and custom headers
• Implements all read operations (GetFile, Exists, Size, GetFileMetadata, etc.) using HTTP HEAD/GET requests
• Provides comprehensive test coverage with mock HTTP server for various scenarios

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
support/datastore/http.go	Core HTTP datastore implementation with read-only operations
support/datastore/http_test.go	Comprehensive test suite with mock HTTP server
support/datastore/datastore.go	Integration point adding HTTP type to datastore factory

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

The HTTP client lacks important security configurations. Consider adding MaxIdleConns, MaxIdleConnsPerHost, and IdleConnTimeout to prevent connection pool exhaustion, and disable automatic redirect following for better security control.

Suggested change

	client := &http.Client{
	Timeout: timeout,
	transport := &http.Transport{
	MaxIdleConns: 100,
	MaxIdleConnsPerHost: 10,
	IdleConnTimeout: 90 * time.Second,
	}
	client := &http.Client{
	Timeout: timeout,
	Transport: transport,
	CheckRedirect: func(req *http.Request, via []*http.Request) error {
	return http.ErrUseLastResponse
	},

[Copilot]

Logging the full error message may expose sensitive information like internal URLs or authentication details. Consider sanitizing the error message before logging or reducing the log level.

Suggested change

	log.Debugf("Error retrieving file '%s': %v", filePath, err)
	log.Debugf("Error retrieving file '%s'", filePath)

[sreuland]

It would be convenient to provide some some comment docs on this method to describe the expected toml datastore config params structure such as for the keys expected like 'timeout' and headers.

[overcat]

ad73020

There are no similar comments in s3 and gcp either. If you would like me to add comments for them in this PR as well, please let me know, thank you.

[miguelnietoa]

Hi there, I'm building a custom indexer and this PR would be useful for my use case. Any updates on getting this merged?