Skip to content

chore: bump otel_collector image version to v1782332626135 - #1161

Open
ssuday wants to merge 1 commit into
awslabs:mainfrom
ssuday:main
Open

chore: bump otel_collector image version to v1782332626135#1161
ssuday wants to merge 1 commit into
awslabs:mainfrom
ssuday:main

Conversation

@ssuday

@ssuday ssuday commented Jul 1, 2026

Copy link
Copy Markdown

Purpose

Bump the HyperPod OTel collector image to include the awsattributelimit processor which enforces a 150-label cap on AMP metrics.

Changes

  • Updated otel_collector image version from v1754424030352 to v1782332626135 in install_otel_collector.sh

Test Plan

Environment:

  • AWS Service: SageMaker HyperPod (EKS)
  • Instance type: EKS cluster (us-west-2)
  • Number of nodes: 4 (verified via node-exporter pod count)

Test commands:

# Created EKS cluster and deployed addon v1.0.7-eksbuild.1
# Verified collector pods running with new image
kubectl get pods -n hyperpod-observability -o wide
kubectl describe pod hyperpod-observability-node-collector-85cjm -n hyperpod-observability

# Injected test metrics via OTLP HTTP receiver
kubectl port-forward hyperpod-observability-node-collector-85cjm 4318:4318 -n hyperpod-observability &
python3 send_test_metrics.py

# Queried AMP to verify label enforcement
awscurl --service aps "https://<amp-endpoint>/api/v1/query?query=test_over_150_labels"

@ssuday
ssuday requested a review from a team as a code owner July 1, 2026 16:53
@ssuday

ssuday commented Jul 1, 2026

Copy link
Copy Markdown
Author

Before merging will do a sanity test with the updated lifecyclescript

@ssuday

ssuday commented Jul 1, 2026

Copy link
Copy Markdown
Author

Additional Testing on the Update LCS:

Created a Cluster using the LCS which has the updated Collector Image

#
# docker ps | grep otel
52421e2801ef   602401143452.dkr.ecr.us-west-2.amazonaws.com/hyperpod/otel_collector:v1782332626135   "/app/otelcollector …"   9 minutes ago   Up 9 minutes             otel-collector
#
# docker inspect otel-collector | grep Image | head -2
        "Image": "sha256:2c3f0f5ea22a3907bed0cfdd93ca7180840b13848263807334b0beebd31aed96",
            "Image": "602401143452.dkr.ecr.us-west-2.amazonaws.com/hyperpod/otel_collector:v1782332626135",
#

Logs of the Collector

Removed few id;s in the below logs

# docker logs otel-collector 2>&1 | grep -v "dcgm_exporter" | tail -20
2026-07-01T19:53:45.177Z	info	prometheusremotewriteexporter/exporter.go:222	starting prometheus remote write exporter	{"resource": {"service.instance.id": "", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "prometheusremotewrite", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "ProtoMsg": "prometheus.WriteRequest"}
2026-07-01T19:53:45.192Z	info	service/service.go:250	Starting otelcol-custom...	{"resource": {"service.instance.id": "", "service.name": "otelcol-custom", "service.version": ""}, "Version": "", "NumCPU": 2}
2026-07-01T19:53:45.192Z	info	extensions/extensions.go:40	Starting extensions...	{"resource": {"service.instance.id": "", "service.name": "otelcol-custom", "service.version": ""}}
2026-07-01T19:53:45.193Z	info	extensions/extensions.go:44	Extension is starting...	{"resource": {"service.instance.id": "", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "sigv4auth", "otelcol.component.kind": "extension"}
2026-07-01T19:53:45.193Z	info	extensions/extensions.go:61	Extension started.	{"resource": {"service.instance.id": "", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "sigv4auth", "otelcol.component.kind": "extension"}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant