spot to spot consolidation is disabled

[hiren95]

karpenter version : 1.0.5
kubernetes version(eks cluster) : 1.31
I configured an EKS cluster and Karpenter using Terraform. After upgrading Karpenter to version 1.0.5, I experienced application downtime due to old nodes failing to drain properly. This prevented new nodes from provisioning, resulting in errors. Although I had enabled consolidation and spot-to-spot, and used node pools and node class configurations, the old nodes still failed to drain. The following error message pertains to an old node that did not drain

Type Reason Age From Message

---

Normal Unconsolidatable 7m47s (x338 over 3d13h) karpenter SpotToSpotConsolidation is disabled, can't replace a spot node with a spot node
Normal NodeNotReady 7s node-controller Node ip-10-172-40-3.ap-south-1.compute.internal status is now: NodeNotReady

[rschalo]

Can you please post your Karpenter deployment spec and NodePools?

[hiren95]

nodepool-service configuration

apiVersion: karpenter.sh/v1beta1
#apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: service
spec:
  template:
    metadata:
    # Labels are arbitrary key-values that are applied to all nodes
    #      labels:
    #        karpenter.sh/provisioner-name: "service"

    # Annotations are arbitrary key-values that are applied to all nodes
    #      annotations:
    #        karpenter.sh/provisioner-name: "service"
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: aws

#      kubeletConfiguration:
#        clusterDNS:
#          - "x.x.x.x" # NodeLocalDNS
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: [ "spot" ]
        - key: "node.kubernetes.io/instance-type"
          operator: In
          values:
            %{ for instance in split(",", service_instance_types) }
            - ${instance}
            %{ endfor }
      #      taints:
      #        - effect: NoSchedule
      #          key: workload-type/service
      #          value: "true"
      disruption:
        consolidationPolicy: WhenUnderutilized
        consolidateAfter: 2m
        expireAfter: 24h
    consolidation:
      enabled: true
      spotToSpot: true
    ttlSecondsAfterEmpty: 30

nodeclass-

apiVersion: karpenter.k8s.aws/v1beta1
#apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
name: aws
spec:
kubelet:
clusterDNS: [ "x.x.x.x" ]
amiFamily: AL2

Optional, configures IMDS for the instance

metadataOptions:
httpPutResponseHopLimit: 2

Optional, IAM instance profile to use for the node identity.

Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes

instanceProfile: "${instance_profile}"

Required, discovers security groups to attach to instances

Each term in the array of securityGroupSelectorTerms is ORed together

Within a single term, all conditions are ANDed

securityGroupSelectorTerms:
# Select on any security group that has both the "karpenter.sh/discovery: {CLUSTER_NAME}" tag
# AND the "environment: test" tag OR any security group with the "my-security-group" name
# OR any security group with ID "sg-063d7acfb4b06c82c"
- tags:
# karpenter.sh/discovery/${cluster_name}: "owned"
kubernetes.io/cluster/${cluster_name}: "owned"

    # environment: test
# - name: my-security-group

Required, discovers subnets to attach to instances

Each term in the array of subnetSelectorTerms is ORed together

Within a single term, all conditions are ANDed

subnetSelectorTerms:
# Select on any subnet that has the "karpenter.sh/discovery: {CLUSTER_NAME}"
# AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
- tags:
# karpenter.sh/discover/${cluster_name}: "true"
karpenter.sh/discovery/${cluster_name}: "true"
# - id: subnet-09fa4a0a8f233a921

blockDeviceMappings:
- deviceName: /dev/xvda
ebs:
volumeSize: 50Gi
volumeType: gp3
encrypted: true

[hiren95]

which type of information you required

[hiren95]

Would you be interested in scheduling a call to discuss this issue in more detail?

[rschalo]

Just looking to see if the Karpenter deployment itself has the correct env for FEATURE_GATES set.

[hiren95]

karpetner deployment

apiVersion: apps/v1
kind: Deployment
metadata:
name: karpenter
namespace: karpenter
uid: 7cfe9951-2ef1-4c0a-8fd6-0f6ed37dfdf1b22a
resourceVersion: '166776630'
generation: 2
creationTimestamp: '2024-12-06T11:17:27Z'
labels:
app.kubernetes.io/instance: karpenter
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: karpenter
app.kubernetes.io/version: 1.0.5
helm.sh/chart: karpenter-1.0.5
k8slens-edit-resource-version: v1
annotations:
deployment.kubernetes.io/revision: '2'
meta.helm.sh/release-name: karpenter
meta.helm.sh/release-namespace: karpenter
managedFields:
- manager: terraform-provider-helm_v2.16.1_x5
operation: Update
apiVersion: apps/v1
time: '2024-12-06T11:17:27Z'
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:meta.helm.sh/release-name: {}
f:meta.helm.sh/release-namespace: {}
f:labels:
.: {}
f:app.kubernetes.io/instance: {}
f:app.kubernetes.io/managed-by: {}
f:app.kubernetes.io/name: {}
f:app.kubernetes.io/version: {}
f:helm.sh/chart: {}
f:spec:
f:progressDeadlineSeconds: {}
f:replicas: {}
f:revisionHistoryLimit: {}
f:selector: {}
f:strategy:
f:rollingUpdate:
.: {}
f:maxSurge: {}
f:maxUnavailable: {}
f:type: {}
f:template:
f:metadata:
f:labels:
.: {}
f:app.kubernetes.io/instance: {}
f:app.kubernetes.io/name: {}
f:spec:
f:affinity:
.: {}
f:nodeAffinity:
.: {}
f:requiredDuringSchedulingIgnoredDuringExecution: {}
f:podAntiAffinity:
.: {}
f:requiredDuringSchedulingIgnoredDuringExecution: {}
f:containers:
k:{"name":"controller"}:
.: {}
f:env:
.: {}
k:{"name":"BATCH_IDLE_DURATION"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"BATCH_MAX_DURATION"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"CLUSTER_ENDPOINT"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"CLUSTER_NAME"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"DISABLE_WEBHOOK"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"FEATURE_GATES"}:
.: {}
f:name: {}
k:{"name":"HEALTH_PROBE_PORT"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"KARPENTER_SERVICE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"KUBERNETES_MIN_VERSION"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"LOG_ERROR_OUTPUT_PATHS"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"LOG_LEVEL"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"LOG_OUTPUT_PATHS"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"MEMORY_LIMIT"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:resourceFieldRef: {}
k:{"name":"METRICS_PORT"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"RESERVED_ENIS"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"SYSTEM_NAMESPACE"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:fieldRef: {}
k:{"name":"VM_MEMORY_OVERHEAD_PERCENT"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"WEBHOOK_METRICS_PORT"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"WEBHOOK_PORT"}:
.: {}
f:name: {}
f:value: {}
f:image: {}
f:imagePullPolicy: {}
f:livenessProbe:
.: {}
f:failureThreshold: {}
f:httpGet:
.: {}
f:path: {}
f:port: {}
f:scheme: {}
f:initialDelaySeconds: {}
f:periodSeconds: {}
f:successThreshold: {}
f:timeoutSeconds: {}
f:name: {}
f:ports:
.: {}
k:{"containerPort":8001,"protocol":"TCP"}:
.: {}
f:containerPort: {}
f:name: {}
f:protocol: {}
k:{"containerPort":8080,"protocol":"TCP"}:
.: {}
f:containerPort: {}
f:name: {}
f:protocol: {}
k:{"containerPort":8081,"protocol":"TCP"}:
.: {}
f:containerPort: {}
f:name: {}
f:protocol: {}
k:{"containerPort":8443,"protocol":"TCP"}:
.: {}
f:containerPort: {}
f:name: {}
f:protocol: {}
f:readinessProbe:
.: {}
f:failureThreshold: {}
f:httpGet:
.: {}
f:path: {}
f:port: {}
f:scheme: {}
f:initialDelaySeconds: {}
f:periodSeconds: {}
f:successThreshold: {}
f:timeoutSeconds: {}
f:resources: {}
f:securityContext:
.: {}
f:allowPrivilegeEscalation: {}
f:capabilities:
.: {}
f:drop: {}
f:readOnlyRootFilesystem: {}
f:runAsGroup: {}
f:runAsNonRoot: {}
f:runAsUser: {}
f:seccompProfile:
.: {}
f:type: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:dnsPolicy: {}
f:nodeSelector: {}
f:priorityClassName: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext:
.: {}
f:fsGroup: {}
f:serviceAccount: {}
f:serviceAccountName: {}
f:terminationGracePeriodSeconds: {}
f:tolerations: {}
f:topologySpreadConstraints:
.: {}
k:{"topologyKey":"topology.kubernetes.io/zone","whenUnsatisfiable":"DoNotSchedule"}:
.: {}
f:labelSelector: {}
f:maxSkew: {}
f:topologyKey: {}
f:whenUnsatisfiable: {}
- manager: node-fetch
operation: Update
apiVersion: apps/v1
time: '2025-02-10T12:14:51Z'
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:labels:
f:k8slens-edit-resource-version: {}
f:spec:
f:template:
f:spec:
f:containers:
k:{"name":"controller"}:
f:env:
k:{"name":"FEATURE_GATES"}:
f:value: {}
- manager: kube-controller-manager
operation: Update
apiVersion: apps/v1
time: '2025-02-10T12:15:25Z'
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:deployment.kubernetes.io/revision: {}
f:status:
f:availableReplicas: {}
f:conditions:
.: {}
k:{"type":"Available"}:
.: {}
f:lastTransitionTime: {}
f:lastUpdateTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"Progressing"}:
.: {}
f:lastTransitionTime: {}
f:lastUpdateTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
f:observedGeneration: {}
f:readyReplicas: {}
f:replicas: {}
f:updatedReplicas: {}
subresource: status
selfLink: /apis/apps/v1/namespaces/karpenter/deployments/karpenter
status:
observedGeneration: 2
replicas: 2
updatedReplicas: 2
readyReplicas: 2
availableReplicas: 2
conditions:
- type: Available
status: 'True'
lastUpdateTime: '2024-12-06T11:17:47Z'
lastTransitionTime: '2024-12-06T11:17:47Z'
reason: MinimumReplicasAvailable
message: Deployment has minimum availability.
- type: Progressing
status: 'True'
lastUpdateTime: '2025-02-10T12:15:25Z'
lastTransitionTime: '2024-12-06T11:17:27Z'
reason: NewReplicaSetAvailable
message: ReplicaSet "karpenter-5fc5dfc496" has successfully progressed.
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/instance: karpenter
app.kubernetes.io/name: karpenter
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/instance: karpenter
app.kubernetes.io/name: karpenter
spec:
containers:
- name: controller
image: >-
public.ecr.aws/karpenter/controller:1.0.5@xxxxxxxxxxxxxxxxxxxx
ports:
- name: http-metrics
containerPort: 8080
protocol: TCP
- name: webhook-metrics
containerPort: 8001
protocol: TCP
- name: https-webhook
containerPort: 8443
protocol: TCP
- name: http
containerPort: 8081
protocol: TCP
env:
- name: KUBERNETES_MIN_VERSION
value: 1.19.0-0
- name: KARPENTER_SERVICE
value: karpenter
- name: WEBHOOK_PORT
value: '8443'
- name: WEBHOOK_METRICS_PORT
value: '8001'
- name: DISABLE_WEBHOOK
value: 'false'
- name: LOG_LEVEL
value: info
- name: LOG_OUTPUT_PATHS
value: stdout
- name: LOG_ERROR_OUTPUT_PATHS
value: stderr
- name: METRICS_PORT
value: '8080'
- name: HEALTH_PROBE_PORT
value: '8081'
- name: SYSTEM_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: MEMORY_LIMIT
valueFrom:
resourceFieldRef:
containerName: controller
resource: limits.memory
divisor: '0'
- name: FEATURE_GATES
value: SpotToSpotConsolidation=true
- name: BATCH_MAX_DURATION
value: 10s
- name: BATCH_IDLE_DURATION
value: 1s
- name: CLUSTER_NAME
value: viking-uat-cluster
- name: CLUSTER_ENDPOINT
value: >-
https://A87BB4A5F6D0873F2fdf1F1192B0F3656565656BEE76.gr7.ap-south-1.eks.amazonaws.com
- name: VM_MEMORY_OVERHEAD_PERCENT
value: '0.075'
- name: RESERVED_ENIS
value: '0'
resources: {}
livenessProbe:
httpGet:
path: /healthz
port: http
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 30
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /readyz
port: http
scheme: HTTP
initialDelaySeconds: 5
timeoutSeconds: 30
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
securityContext:
capabilities:
drop:
- ALL
runAsUser: 65532
runAsGroup: 65532
runAsNonRoot: true
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
seccompProfile:
type: RuntimeDefault
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: karpenter
serviceAccount: karpenter
securityContext:
fsGroup: 65532
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: karpenter.sh/nodepool
operator: DoesNotExist
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/instance: karpenter
app.kubernetes.io/name: karpenter
topologyKey: kubernetes.io/hostname
schedulerName: default-scheduler
tolerations:
- key: CriticalAddonsOnly
operator: Exists
priorityClassName: system-cluster-critical
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app.kubernetes.io/instance: karpenter
app.kubernetes.io/name: karpenter
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 25%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600

[hiren95]

Just looking to see if the Karpenter deployment itself has the correct env for FEATURE_GATES set.

i have shared deployment yaml, can you review it