kafka pod crashes on amd64 kind cluster

[stytchiz]

Bug Report

Which version of the demo you are using?

c281f7c

Symptom

The kafka pod is crashing on a cluster built using kind.

$ kubectl get pods
...
opentelemetry-demo-imageprovider-7466d894fb-djb9p           1/1     Running            0               7m11s
opentelemetry-demo-jaeger-7785549bb-8krc8                   1/1     Running            2 (5h43m ago)   2d20h
opentelemetry-demo-kafka-76d4d9f48b-9bwms                   0/1     CrashLoopBackOff   6 (68s ago)     7m11s
opentelemetry-demo-loadgenerator-6994f5db8-v9hnx            1/1     Running            0               7m10s
opentelemetry-demo-paymentservice-857974bcdb-lvbnk          1/1     Running            0               7m10s
...

$ kubectl logs opentelemetry-demo-kafka-76d4d9f48b-9bwms
exec /__cacert_entrypoint.sh: exec format error

Investigating this error further, apparently it could have something to do with the image being built for arm64 but running the container on amd64. I'm not using a M1 Macbook (x86 Linux) so can not use the arm64-kind binary.

$ docker images
REPOSITORY     TAG       IMAGE ID       CREATED       SIZE
kindest/node   <none>    18fbefc20a71   4 weeks ago   1.49GB
$ docker image inspect 18fbefc20a71
[
    {
        "Id": "sha256:18fbefc20a7113353c7b75b5c869d7145a6abd6269154825872dc59c1329912e",
        "RepoTags": [],
        "RepoDigests": [
            "kindest/node@sha256:18fbefc20a7113353c7b75b5c869d7145a6abd6269154825872dc59c1329912e"
        ],
        ...
        "Architecture": "amd64",
        "Os": "linux",
    }
]

What is the expected behavior?

See above.

What is the actual behavior?

See above.

Reproduce

1. Install kind-linux-amd64 per https://kind.sigs.k8s.io/docs/user/quick-start/#installing-from-release-binaries
2. https://opentelemetry.io/docs/demo/kubernetes-deployment/#install-using-helm-recommended
3. kubectl get pods

Additional Context

Please feel free to add any other context about the problem here.

[julianocosta89]

@stytchiz do you happen to know why the ARM64 container is being pulled instead of the AMD64?
We have multi arch releases for all containers, Kafka included:

https://hub.docker.com/layers/otel/demo/latest-kafka/images/sha256-da74d35133252c247b281c7ee54339cc8689b9987fc61ce271c0e6bb04200b9e?context=explore

[stytchiz]

Hi! No I don't.

I'm using https://hub.docker.com/layers/otel/demo/1.12.0-kafka/images/sha256-9878e8959b942b9b30ce97f479e08bc1bd1faa37623b1b7d8cc5ff73f8b40979?context=explore.

From the pod description output,

Containers:
  kafka:
    Container ID:   containerd://2be5ea6fdcd2af811fdd4f5774c49e8cfcaa6d508e6c2c3d08275d6640ec1f79
    Image:          ghcr.io/open-telemetry/demo:1.12.0-kafka
    Image ID:       ghcr.io/open-telemetry/demo@sha256:071a788162e84bbb7a614aaf941824a44df691be6263a26bd539bcc35d5d2dd7
    Ports:          9092/TCP, 9093/TCP

[stytchiz]

I tried specifying the amd64 manifest digest, but still get the same error.

$ k describe deployments opentelemetry-demo-kafka
  Containers:
   kafka:
    Image:       ghcr.io/open-telemetry/demo:1.12.0-kafka@sha256:9878e8959b942b9b30ce97f479e08bc1bd1faa37623b1b7d8cc5ff73f8b40979
    Ports:       9092/TCP, 9093/TCP
    Host Ports:  0/TCP, 0/TCP