AWS ap-southeast-5 location constraint is incompatible

[khanhdd-ahs]

Thanos, Prometheus and Golang version used:
thanos:0.37.2

Object Storage Provider:
S3
What happened:
thanos-storegateway-0 pod is going in crashed state
What you expected to happen:
thanos-storegateway-0 pod should come up
How to reproduce it (as minimally and precisely as possible):
Deploy thanos with S3 bucket in region ap-southeast-5 (Malaysia)
Full logs to relevant components:

ts=2025-03-06T04:28:42.337383574Z caller=store.go:468 level=info msg="initializing bucket store"
ts=2025-03-06T04:29:12.814453113Z caller=intrumentation.go:67 level=warn msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: The ap-southeast-5 location constraint is incompatible for the region specific endpoint this request was sent to."
ts=2025-03-06T04:29:12.814497519Z caller=http.go:91 level=info service=http/server component=store msg="internal server is shutting down" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: The ap-southeast-5 location constraint is incompatible for the region specific endpoint this request was sent to."
ts=2025-03-06T04:29:12.814574919Z caller=intrumentation.go:56 level=info msg="changing probe status" status=ready
ts=2025-03-06T04:29:12.814670543Z caller=grpc.go:167 level=info service=gRPC/server component=store msg="listening for serving gRPC" address=0.0.0.0:10901
ts=2025-03-06T04:29:12.814772444Z caller=http.go:110 level=info service=http/server component=store msg="internal server is shutdown gracefully" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: The ap-southeast-5 location constraint is incompatible for the region specific endpoint this request was sent to."
ts=2025-03-06T04:29:12.814791267Z caller=intrumentation.go:81 level=info msg="changing probe status" status=not-healthy reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: The ap-southeast-5 location constraint is incompatible for the region specific endpoint this request was sent to."
ts=2025-03-06T04:29:12.81483647Z caller=intrumentation.go:67 level=warn msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: The ap-southeast-5 location constraint is incompatible for the region specific endpoint this request was sent to."
ts=2025-03-06T04:29:12.814885824Z caller=grpc.go:174 level=info service=gRPC/server component=store msg="internal server is shutting down" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: The ap-southeast-5 location constraint is incompatible for the region specific endpoint this request was sent to."
ts=2025-03-06T04:29:12.814923741Z caller=grpc.go:187 level=info service=gRPC/server component=store msg="gracefully stopping internal server"
ts=2025-03-06T04:29:12.815030393Z caller=grpc.go:200 level=info service=gRPC/server component=store msg="internal server is shutdown gracefully" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: The ap-southeast-5 location constraint is incompatible for the region specific endpoint this request was sent to."
ts=2025-03-06T04:29:12.815197977Z caller=main.go:171 level=error err="The ap-southeast-5 location constraint is incompatible for the region specific endpoint this request was sent to.\nBaseFetcher: iter bucket\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetchMetadata\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:526\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch.func2\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:597\ngithub.com/golang/groupcache/singleflight.(*Group).Do\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/pkg/mod/github.com/golang/[email protected]/singleflight/singleflight.go:56\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:595\ngithub.com/thanos-io/thanos/pkg/block.(*MetaFetcher).Fetch\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/pkg/block/fetcher.go:657\ngithub.com/thanos-io/thanos/pkg/store.(*BucketStore).SyncBlocks\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/pkg/store/bucket.go:659\ngithub.com/thanos-io/thanos/pkg/store.(*BucketStore).InitialSync\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/pkg/store/bucket.go:726\nmain.runStore.func6.1\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/cmd/thanos/store.go:477\ngithub.com/thanos-io/thanos/pkg/runutil.RetryWithLog\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:114\ngithub.com/thanos-io/thanos/pkg/runutil.Retry\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:104\nmain.runStore.func6\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/cmd/thanos/store.go:476\ngithub.com/oklog/run.(*Group).Run.func1\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/pkg/mod/github.com/oklog/[email protected]/group.go:38\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_arm64.s:1223\nsync block\ngithub.com/thanos-io/thanos/pkg/store.(*BucketStore).InitialSync\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/pkg/store/bucket.go:727\nmain.runStore.func6.1\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/cmd/thanos/store.go:477\ngithub.com/thanos-io/thanos/pkg/runutil.RetryWithLog\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:114\ngithub.com/thanos-io/thanos/pkg/runutil.Retry\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:104\nmain.runStore.func6\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/cmd/thanos/store.go:476\ngithub.com/oklog/run.(*Group).Run.func1\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/pkg/mod/github.com/oklog/[email protected]/group.go:38\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_arm64.s:1223\nbucket store initial sync\nmain.runStore.func6\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/cmd/thanos/store.go:482\ngithub.com/oklog/run.(*Group).Run.func1\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/pkg/mod/github.com/oklog/[email protected]/group.go:38\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_arm64.s:1223\nstore command failed\nmain.main\n\t/bitnami/blacksmith-sandox/thanos-0.37.2/src/github.com/thanos-io/thanos/cmd/thanos/main.go:171\nruntime.main\n\t/opt/bitnami/go/src/runtime/proc.go:272\nruntime.goexit\n\t/opt/bitnami/go/src/runtime/asm_arm64.s:1223"

Anything else we need to know:
I have created an issue to add support for this region in MinIO.
minio/minio-go#2073

[dosubot]

Hello, @khanhdd-ahs! I'm here to help you with any bugs, questions, or contributions you have. Let's work together to solve your issue.

The error you're encountering is related to the AWS S3 location constraint for the ap-southeast-5 region, which is causing the Thanos store gateway pod to crash. This issue arises because the MinIO Go client, used by Thanos, may not yet support this new AWS region.

To address this, you can manually specify or override the AWS S3 endpoint in your Thanos configuration to ensure compatibility. Here's how you can configure it:

type: S3
config:
  bucket: "your-bucket-name"
  endpoint: "s3.ap-southeast-5.amazonaws.com"
  access_key: "your-access-key"
  secret_key: "your-secret-key"
  region: "ap-southeast-5"
  aws_sdk_auth: true  # Use AWS SDK for authentication
  signature_version2: false  # Ensure signature v4 is used

This configuration explicitly sets the endpoint and region for the ap-southeast-5 region, which should help in resolving the location constraint issue. Additionally, ensure that your Thanos version and dependencies are up-to-date to support the latest AWS regions ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[ben-ganley]

I have tried with the suggested config but the result is the same. The underlying cause is the minio lib doesn't yet support this region.

I think until its fixed in minio and updated in Thanos this will continue to be an issue.