[Bug]: helm upgrade fails on same version on strimzi operator:0.43.0, controller goes in crashloop version: 3.8.0

[sujayran]

Bug Description

kafka-controller-101 0/1 CrashLoopBackOff 11 (3m7s ago) 37m

TOPIC
evnets           kafka     2            2                    True

controller log:

emoved directory '/tmp/hsperfdata_kafka'
removed '/tmp/kafka/strimzi.kafka.metadata.config.state'
removed '/tmp/kafka/clients.truststore.p12'
removed '/tmp/kafka/cluster.keystore.p12'
removed '/tmp/kafka/cluster.truststore.p12'
removed directory '/tmp/kafka'
removed '/tmp/strimzi.properties'
STRIMZI_BROKER_ID=101
Preparing truststore for replication listener
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for replication listener is complete
Looking for the right CA
Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt
Preparing keystore for replication and clienttls listener
Preparing keystore for replication and clienttls listener is complete
Preparing truststore for client authentication
Adding /opt/kafka/client-ca-certs/ca.crt to truststore /tmp/kafka/clients.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for client authentication is complete
Starting Kafka with configuration:
##############################
##############################
# This file is automatically generated by the Strimzi Cluster Operator
# Any changes to this file will be ignored and overwritten!
##############################
##############################

##########
# Node / Broker ID
##########
node.id=101

##########
# Kafka message logs configuration
##########
log.dirs=/var/lib/kafka/data-0/kafka-log101

##########
# Control Plane listener
##########
listener.name.controlplane-9090.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
listener.name.controlplane-9090.ssl.keystore.password=[hidden]
listener.name.controlplane-9090.ssl.keystore.type=PKCS12
listener.name.controlplane-9090.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
listener.name.controlplane-9090.ssl.truststore.password=[hidden]
listener.name.controlplane-9090.ssl.truststore.type=PKCS12
listener.name.controlplane-9090.ssl.client.auth=required

##########
# Common listener configuration
##########
listener.security.protocol.map=CONTROLPLANE-9090:SSL
listeners=CONTROLPLANE-9090://0.0.0.0:9090
sasl.enabled.mechanisms=
ssl.endpoint.identification.algorithm=HTTPS

##########
# User provided configuration
##########
auto.create.topics.enable=false
connections.idle.timeouts.ms=600000
default.replication.factor=3
min.insync.replicas=2
offsets.topic.num.partitions=50
offsets.topic.num.partitions.downscale.limit=3
offsets.topic.num.partitions.upscale.limit=6
offsets.topic.replication.factor=3
transaction.state.log.min.isr=2
transaction.state.log.num.partitions=50
transaction.state.log.replication.factor=3


##########
# KRaft configuration
##########
process.roles=controller
controller.listener.names=CONTROLPLANE-9090
controller.quorum.voters=100@kafka-controller-100.kafka-kafka-brokers.fed-kafka-strimzi.svc.cluster.local:9090,101@kafka-controller-101.kafka-kafka-brokers.fed-kafka-strimzi.svc.cluster.local:9090,102@kafka-controller-102.kafka-kafka-brokers.fed-kafka-strimzi.svc.cluster.local:9090

##########
# KRaft metadata log dir configuration
##########
metadata.log.dir=/var/lib/kafka/data-0/kafka-log101
Kafka metadata config state [4]
Using KRaft [true]
Making sure the Kraft storage is formatted with cluster ID ihmB84a0QJ2B8esD10YC8w and metadata version 3.8-IV0
2025-01-31 20:19:44,362 INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [main]
Formatting /var/lib/kafka/data-0/kafka-log101 with metadata.version 3.8-IV0.
Error while writing meta.properties file /var/lib/kafka/data-0/kafka-log101: /var/lib/kafka/data-0/kafka-log101

Steps to reproduce

helm install -n <fed-name> <fed-name> --create-namespace -f values.yaml <chart-name>

topic parameter
    partitions: 2 
    replicationFactor: 2 
    retentionBytes: "95000000" 
    retentionms: "25920000" 
    segmentBytes: "90000000"

bytes coming in ~2.2 mb/s

helm upgrade -n fed-kafka-strimzi fed-kafka-strimzi -f fed-kafka-strimzi-values.yaml

Values after and before remains same just image tag changes which result in operator upgrade and the controller

some time controller goes in crash loop

Making sure the Kraft storage is formatted with cluster ID ihmB84a0QJ2B8esD10YC8w and metadata version 3.8-IV0
2025-01-31 20:19:44,362 INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [main]
Formatting /var/lib/kafka/data-0/kafka-log101 with metadata.version 3.8-IV0.
Error while writing meta.properties file /var/lib/kafka/data-0/kafka-log101: /var/lib/kafka/data-0/kafka-log101

Expected behavior

Upgrade should perform, all pods should get upgraded in rolling upgrade.

Strimzi version

3.8.0

Kubernetes version

v1.29.11

Installation method

helm

Infrastructure

openstack

Configuration files and logs

kc edit kafka kafka -n

apiVersion: kafka.strimzi.io/v1beta2 
kind: Kafka 
metadata: 
  annotations: 
    meta.helm.sh/release-name: fed-kafka-strimzi 
    meta.helm.sh/release-namespace: fed-kafka-strimzi 
    strimzi.io/kraft: enabled 
    strimzi.io/node-pools: enabled 
  creationTimestamp: "2025-01-31T19:18:20Z" 
  generation: 1 
  labels: 
    app.kubernetes.io/managed-by: Helm 
  name: kafka 
  namespace: fed-kafka-strimzi 
  resourceVersion: "10385013" 
  uid: 0d708517-c34f-48fa-88e8-282fe5af7dbb 
spec: 
  cruiseControl: 
    brokerCapacity: 
      cpu: 2000m 
      inboundNetwork: 9000KB/s 
      outboundNetwork: 9000KB/s 
    config: 
      anomaly.detection.goals: com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal 
      anomaly.detection.interval.ms: 120000 
      anomaly.notifier.class: com.linkedin.kafka.cruisecontrol.detector.notifier.NoopNotifier 
      broker.failure.detection.interval.ms: 60000 
      broker.metric.sample.store.topic: __KafkaCruiseControlModelTrainingSamples 
      broker.metrics.window.ms: 300000 
      broker.sample.store.topic.partition.count: 32 
      completed.user.task.retention.time.ms: 21600000 
      cpu.capacity.threshold: 0.8 
      cruise.control.metrics.topic.num.partitions: 3 
      cruise.control.metrics.topic.replication.factor: 3 
      default.goals: | 
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal 
      demotion.history.retention.time.ms: 86400000 
      disk.capacity.threshold: 0.8 
      goals: | 
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.kafkaassigner.KafkaAssignerDiskUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.PreferredLeaderElectionGoal 
      hard.goals: | 
        com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal 
      max.replicas.per.broker: 10000 
      metric.anomaly.analyzer.metrics: | 
        BROKER_PRODUCE_LOCAL_TIME_MS_MAX,BROKER_PRODUCE_LOCAL_TIME_MS_MEAN,BROKER_CONSUMER_FETCH_LOCAL_TIME_MS_MAX,BROKER_CONSUMER_FETCH_LOCAL_TIME_MS_MEAN,BROKER_FOLLOWER_FETCH_LOCAL_TIME_MS_MAX,BROKER_FOLLOWER_FETCH_LOCAL_TIME_MS_MEAN,BROKER_LOG_FLUSH_TIME_MS_MAX,BROKER_LOG_FLUSH_TIME_MS_MEAN 
      metric.anomaly.finder.class: com.linkedin.kafka.cruisecontrol.detector.KafkaMetricAnomalyFinder 
      metric.anomaly.percentile.lower.threshold: 10 
      metric.anomaly.percentile.upper.threshold: 90 
      metric.sampling.interval.ms: 120000 
      min.samples.per.broker.metrics.window: 1 
      min.samples.per.partition.metrics.window: 1 
      min.valid.partition.ratio: 0.95 
      num.broker.metrics.windows: 20 
      num.concurrent.partition.movements.per.broker: 10 
      num.partition.metrics.windows: 1 
      partition.metric.sample.store.on.execution.topic: __CruiseControlPartitionMetricSamplesDuringExecution 
      partition.metric.sample.store.topic: __KafkaCruiseControlPartitionMetricSamples 
      partition.metrics.window.ms: 300000 
      partition.sample.store.topic.partition.count: 32 
      proposal.expiration.ms: 60000 
      sample.store.topic.replication.factor: 3 
      webserver.http.cors.origin: http://localhost:8080/ 
      webserver.security.enable: false 
      webserver.ssl.enable: false 
    livenessProbe: 
      failureThreshold: 5 
      initialDelaySeconds: 15 
      periodSeconds: 10 
      successThreshold: 1 
      timeoutSeconds: 5 
    logging: 
      loggers: 
        rootLogger.level: WARN 
      type: inline 
    metricsConfig: 
      type: jmxPrometheusExporter 
      valueFrom: 
        configMapKeyRef: 
          key: cc-metrics-config.yml 
          name: cruise-control-metrics 
    readinessProbe: 
      failureThreshold: 5 
      initialDelaySeconds: 15 
      periodSeconds: 10 
      successThreshold: 1 
      timeoutSeconds: 5 
    resources: 
      limits: 
        cpu: 1000m 
        memory: 4Gi 
      requests: 
        cpu: 500m 
        memory: 2Gi
    template:
      pod:
        metadata:
          labels:
            app: cruisecontrol
            kafka_cr: kafka
        priorityClassName: uc-priority-6
  entityOperator:
    topicOperator:
      reconciliationIntervalMs: 60000
      resources:
        limits:
          cpu: "1"
          memory: 512Mi
        requests:
          cpu: "1"
          memory: 512Mi
    userOperator: {}
  kafka:
    config:
      auto.create.topics.enable: "false"
      connections.idle.timeouts.ms: 600000
      default.replication.factor: 3
      min.insync.replicas: 2
      offsets.topic.num.partitions: 50
      offsets.topic.num.partitions.downscale.limit: 3
      offsets.topic.num.partitions.upscale.limit: 6
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.num.partitions: 50
      transaction.state.log.replication.factor: 3
    listeners:
    - name: internal
      port: 29092
      tls: false
      type: internal
    - name: controller
      port: 29093
      tls: false
      type: internal
    - configuration:
        bootstrap: {}
        brokers:
        - broker: 0
          loadBalancerIP: ""
        - broker: 1
          loadBalancerIP: ""
        - broker: 2
          loadBalancerIP: ""
      name: external
      port: 19090
      tls: false
      type: loadbalancer
    logging:
      loggers:
        kafka.root.logger.level: WARN
        log4j.logger.kafka: WARN
        log4j.logger.kafka.authorizer.logger: WARN
        log4j.logger.kafka.log.LogCleaner: WARN
        log4j.logger.org.apache.kafka: WARN
        log4j.logger.org.apache.kafka.controller: WARN
      type: inline
    metadataVersion: 3.8-IV0
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: jmx-exporter-config.yml
          name: kafka-metrics
    version: 3.8.0
status:
  clusterId: ihmB84a0QJ2B8esD10YC8w
  conditions:
  - lastTransitionTime: "2025-01-31T19:45:11.574275083Z"
    message: Pod is unschedulable or is not starting
    reason: FatalProblem
    status: "True"
    type: NotReady
  kafkaMetadataState: KRaft
  kafkaMetadataVersion: 3.8-IV0
  kafkaNodePools:
  - name: broker
  - name: controller
  kafkaVersion: 3.8.0
  observedGeneration: 1
  operatorLastSuccessfulVersion: 0.43.0
  registeredNodeIds:
  - 0
  - 1
  - 2
  - 100
  - 101
  - 102

=======
controller

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  annotations:
    meta.helm.sh/release-name: fed-kafka-strimzi
    meta.helm.sh/release-namespace: fed-kafka-strimzi
    strimzi.io/next-node-ids: '[100-199]'
  creationTimestamp: "2025-01-31T19:18:20Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    strimzi.io/cluster: kafka
  name: controller
  namespace: fed-kafka-strimzi
  resourceVersion: "10368754"
  uid: 60391e17-53e6-4a91-8ce6-437ad00d1d04
spec:
  jvmOptions:
    -Xms: 4g
    -Xmx: 4g
  replicas: 3
  resources:
    limits:
      cpu: 500m
      memory: 2Gi
    requests:
      cpu: 300m
      memory: 2Gi
  roles:
  - controller
  storage:
    type: jbod
    volumes:
    - class: kfsc-strimzi
      deleteClaim: false
      id: 0
      kraftMetadata: shared
      size: 1430Mi
      type: persistent-claim
  template:
    pod:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - kafka
                - key: kafka_cr
                  operator: In
                  values:
                  - kafka
              topologyKey: kubernetes.io/hostname
            weight: 1
      priorityClassName: uc-priority-2
status:
  clusterId: ihmB84a0QJ2B8esD10YC8w
  conditions: []
  labelSelector: strimzi.io/cluster=kafka,strimzi.io/name=kafka-kafka,strimzi.io/kind=Kafka,strimzi.io/pool-name=controller
  nodeIds:
  - 100
  - 101
  - 102
  observedGeneration: 1
  replicas: 3
  roles:
  - controller

======= broker======

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  annotations:
    meta.helm.sh/release-name: fed-kafka-strimzi
    meta.helm.sh/release-namespace: fed-kafka-strimzi
    strimzi.io/next-node-ids: '[0-99]'
  creationTimestamp: "2025-01-31T19:18:20Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    strimzi.io/cluster: kafka
  name: broker
  namespace: fed-kafka-strimzi
  resourceVersion: "10368751"
  uid: 55b24fc4-af0b-422d-8f31-78ed4d1e27bd
spec:
  jvmOptions:
    -Xms: 4g
    -Xmx: 4g
  replicas: 3
  resources:
    limits:
      cpu: 2000m
      memory: 5Gi
    requests:
      cpu: 1000m
      memory: 5Gi
  roles:
  - broker
  storage:
    type: jbod
    volumes:
    - class: kfsc-strimzi
      deleteClaim: false
      id: 0
      kraftMetadata: shared
      size: 4768Mi
      type: persistent-claim
  template:
    pod:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - kafka
                - key: kafka_cr
                  operator: In
                  values:
                  - kafka
              topologyKey: kubernetes.io/hostname
            weight: 1
      metadata:
        labels:
          app: kafka
          kafka_cr: kafka
      priorityClassName: uc-priority-2
status:
  clusterId: ihmB84a0QJ2B8esD10YC8w
  conditions: []
  labelSelector: strimzi.io/cluster=kafka,strimzi.io/name=kafka-kafka,strimzi.io/kind=Kafka,strimzi.io/pool-name=broker
  nodeIds:
  - 0
  - 1
  - 2
  observedGeneration: 1
  replicas: 3
  roles:
  - broker

controller log:

removed directory '/tmp/hsperfdata_kafka'
removed '/tmp/kafka/strimzi.kafka.metadata.config.state'
removed '/tmp/kafka/clients.truststore.p12'
removed '/tmp/kafka/cluster.keystore.p12'
removed '/tmp/kafka/cluster.truststore.p12'
removed directory '/tmp/kafka'
removed '/tmp/strimzi.properties'
STRIMZI_BROKER_ID=101
Preparing truststore for replication listener
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for replication listener is complete
Looking for the right CA
Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt
Preparing keystore for replication and clienttls listener
Preparing keystore for replication and clienttls listener is complete
Preparing truststore for client authentication
Adding /opt/kafka/client-ca-certs/ca.crt to truststore /tmp/kafka/clients.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for client authentication is complete
Starting Kafka with configuration:
##############################
##############################
# This file is automatically generated by the Strimzi Cluster Operator
# Any changes to this file will be ignored and overwritten!
##############################
##############################

##########
# Node / Broker ID
##########
node.id=101

##########
# Kafka message logs configuration
##########
log.dirs=/var/lib/kafka/data-0/kafka-log101

##########
# Control Plane listener
##########
listener.name.controlplane-9090.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
listener.name.controlplane-9090.ssl.keystore.password=[hidden]
listener.name.controlplane-9090.ssl.keystore.type=PKCS12
listener.name.controlplane-9090.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
listener.name.controlplane-9090.ssl.truststore.password=[hidden]
listener.name.controlplane-9090.ssl.truststore.type=PKCS12
listener.name.controlplane-9090.ssl.client.auth=required

##########
# Common listener configuration
##########
listener.security.protocol.map=CONTROLPLANE-9090:SSL
listeners=CONTROLPLANE-9090://0.0.0.0:9090
sasl.enabled.mechanisms=
ssl.endpoint.identification.algorithm=HTTPS

##########
# User provided configuration
##########
auto.create.topics.enable=false
connections.idle.timeouts.ms=600000
default.replication.factor=3
min.insync.replicas=2
offsets.topic.num.partitions=50
offsets.topic.num.partitions.downscale.limit=3
offsets.topic.num.partitions.upscale.limit=6
offsets.topic.replication.factor=3
transaction.state.log.min.isr=2
transaction.state.log.num.partitions=50
transaction.state.log.replication.factor=3


##########
# KRaft configuration
##########
process.roles=controller
controller.listener.names=CONTROLPLANE-9090
controller.quorum.voters=100@kafka-controller-100.kafka-kafka-brokers.fed-kafka-strimzi.svc.cluster.local:9090,101@kafka-controller-101.kafka-kafka-brokers.fed-kafka-strimzi.svc.cluster.local:9090,102@kafka-controller-102.kafka-kafka-brokers.fed-kafka-strimzi.svc.cluster.local:9090

##########
# KRaft metadata log dir configuration
##########
metadata.log.dir=/var/lib/kafka/data-0/kafka-log101
Kafka metadata config state [4]
Using KRaft [true]
Making sure the Kraft storage is formatted with cluster ID ihmB84a0QJ2B8esD10YC8w and metadata version 3.8-IV0
2025-01-31 20:19:44,362 INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [main]
Formatting /var/lib/kafka/data-0/kafka-log101 with metadata.version 3.8-IV0.
Error while writing meta.properties file /var/lib/kafka/data-0/kafka-log101: /var/lib/kafka/data-0/kafka-log101

Additional context

No response

[scholzj]

Please make sure everything is properly formatted. Otherwise it is not readable.

[sujayran]

Please let me know if its readable

[scholzj]

This seems like some storage issue to me. Do you have free disk space? Is the storage readable and writeable?

The steps to reproduce do not seem really clear so I cannot really try it. But the Kafka logs points to some kind of storage problem.

[sujayran]

thanks for reply, i am using jbod option for borker and controller

pvc-10377b04-c620-472f-9a14-96e7bc8c3d9f           4768Mi     RWO            Delete           Bound    fed-kafka-strimzi/data-0-kafka-broker-2                              kfsc-strimzi        <unset>                          23h
pvc-65b34428-5fa3-4235-a6f8-6050ed9c4489           4768Mi     RWO            Delete           Bound    fed-kafka-strimzi/data-0-kafka-broker-1                              kfsc-strimzi        <unset>                          23h
pvc-90c7722f-3bcd-4b32-a64e-316d68f1b2f1           1430Mi     RWO            Delete           Bound    fed-kafka-strimzi/data-0-kafka-controller-101                        kfsc-strimzi        <unset>                          23h
pvc-be3d1b2e-bb95-4f9c-aa4f-e90801cdc025           4768Mi     RWO            Delete           Bound    fed-kafka-strimzi/data-0-kafka-broker-0                              kfsc-strimzi        <unset>                          23h
pvc-d3e14c8d-a3ab-40ba-b45b-0ef6890c9e17           1430Mi     RWO            Delete           Bound    fed-kafka-strimzi/data-0-kafka-controller-100                        kfsc-strimzi        <unset>                          23h
pvc-d971b09c-73f3-4213-aba9-3b7ed5d421ce           1430Mi     RWO            Delete           Bound    fed-kafka-strimzi/data-0-kafka-controller-102                        kfsc-strimzi        <unset>                          23h

and PVC

NAME                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
data-0-kafka-broker-0         Bound    pvc-be3d1b2e-bb95-4f9c-aa4f-e90801cdc025   4768Mi     RWO            kfsc-strimzi   <unset>                 23h
data-0-kafka-broker-1         Bound    pvc-65b34428-5fa3-4235-a6f8-6050ed9c4489   4768Mi     RWO            kfsc-strimzi   <unset>                 23h
data-0-kafka-broker-2         Bound    pvc-10377b04-c620-472f-9a14-96e7bc8c3d9f   4768Mi     RWO            kfsc-strimzi   <unset>                 23h
data-0-kafka-controller-100   Bound    pvc-d3e14c8d-a3ab-40ba-b45b-0ef6890c9e17   1430Mi     RWO            kfsc-strimzi   <unset>                 23h
data-0-kafka-controller-101   Bound    pvc-90c7722f-3bcd-4b32-a64e-316d68f1b2f1   1430Mi     RWO            kfsc-strimzi   <unset>                 23h
data-0-kafka-controller-102   Bound    pvc-d971b09c-73f3-4213-aba9-3b7ed5d421ce   1430Mi     RWO            kfsc-strimzi   <unset>                 23h

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    meta.helm.sh/release-name: fed-kafka-strimzi
    meta.helm.sh/release-namespace: fed-kafka-strimzi
  creationTimestamp: "2025-01-31T21:19:45Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: kfsc-strimzi
  resourceVersion: "10439118"
  uid: d5981dc9-4852-4a10-9bf8-be755d5e77ac
provisioner: rancher.io/local-path
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
~      

The steps to reproduce is used

helm install -n fed-kafka-strimzi fed-kafka-strimzi --create-namespace -f <values>.ymal <chart-name>

with seprate controller / broker in kafkaNodePool

kafka-broker-0                               1/1     Running   0              23h
kafka-broker-1                               1/1     Running   0              23h
kafka-broker-2                               1/1     Running   0              23h
kafka-controller-100                         1/1     Running   0     23h
kafka-controller-101                         1/1     Running   0      23h
kafka-controller-102                         1/1     Running   0    23h
kafka-cruise-control-854b5984c7-48dd5        1/1     Running   0              23h
kafka-entity-operator-5bc4c77dbd-znqwg       2/2     Running   0              23h
mirrormaker-mirrormaker2-0                   1/1     Running   2 (23h ago)    23h
pod-kafka-strimzi-operator-c5cb65565-ng5rb   1/1     Running   0              23h
prometheus-deployment-56dd5d4dcd-lxj7f       1/1     Running   0              23h

start producing data to the installed kafka server kafka_server_brokertopicmetrics_bytesin_total 2.5 mb in 5 ims

helm upgrade -n fed-kafka-strimzi fed-kafka-strimzi -f <same-value>.yaml <new chart with strimzi version>

upgrade starts fine, with operator getting rolling upgraded and the controller starts and then goes in crash back loop, with below errror

metadata.log.dir=/var/lib/kafka/data-0/kafka-log101
Kafka metadata config state [4]
Using KRaft [true]
Making sure the Kraft storage is formatted with cluster ID ihmB84a0QJ2B8esD10YC8w and metadata version 3.8-IV0
2025-01-31 20:19:44,362 INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [main]
Formatting /var/lib/kafka/data-0/kafka-log101 with metadata.version 3.8-IV0.
Error while writing meta.properties file /var/lib/kafka/data-0/kafka-log101: /var/lib/kafka/data-0/kafka-log101

Also upfter this point it not recoverable as well. :(

sample controller yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controller
  labels:
    strimzi.io/cluster: kafka
  annotations:
    strimzi.io/next-node-ids: "[100-199]"
spec:
  replicas: {{ .Values.controller.count }}
  template:
      pod:
        priorityClassName: ""
        affinity:
          {{- if (eq .Values.global.deployment.nodeDataplaneAffinity "true") }}
          nodeAffinity:
            {{ toYaml .Values.affinity.nodeAffinity | nindent 12 }}
          {{- end }}
          podAntiAffinity:
          {{ toYaml .Values.affinity.podAntiAffinity | nindent 12 }}
  resources:
      limits:
        memory: {{ .Values.controller.resources.limits.memory }}
        cpu: {{ .Values.controller.resources.limits.cpu }}
      requests:
        memory: {{ .Values.controller.resources.requests.memory }}
        cpu: {{ .Values.controller.resources.requests.cpu }}
  jvmOptions:
      -Xms: {{ .Values.broker.JVMOpts.xms }}
      -Xmx: {{ .Values.broker.JVMOpts.xmx }}
  roles:
    - controller
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 1430Mi
        kraftMetadata: shared
        deleteClaim: false
        class: {{ .Values.storage.storageclass }}

[scholzj]

You don't use JBOD. You have just one disk in the configs you shared. Also, that does not help you in any way if you run out of disk space. Plus controllers would use only one disk even if you configure it with multiple disks.

I think your steps are also still unclear. What helm chart are you reffering to? What versions are you upgrading between?

[sujayran]

Thanks for quick reply, Ah it's internal helm charts created with strimzi 3.8 yamls and crds. Version of strimzi is not changing. It's the internal charts build number tag.

Why you think the formatting error can occur ? and do we have any steps to recover Or check if that could be contributing to this. We had enough disk space available when this happened. Also it happens every time.

[scholzj]

Right, so ... that is the point I'm trying to make. Nobody can reproduce anything from the commands using your internal Helm Chart. You have to share the exact steps and the exact YAMLs.