Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Schedules in budgets are ignored completely #1939

Open
nantiferov opened this issue Jan 28, 2025 · 1 comment
Open

Schedules in budgets are ignored completely #1939

nantiferov opened this issue Jan 28, 2025 · 1 comment
Labels
kind/bug Categorizes issue or PR as related to a bug. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@nantiferov
Copy link

Description

Observed Behavior:

Hi,
We're running Karpenter 1.0.8 in multiple EKS clusters and noticed that schedule in disruption.budgets is ignored completely. We have different configurations for disruption.budgets and all of them act like schedule doesn't exist there.

We weren't using schedule in 0.37, so I cannot say if this is a bug in v1.0 or schedule was never working.
There are also no errors in Karpenter logs itself and all functionality is working fine without issues, except schedule.

Expected Behavior:

Schedule section in disruption.budgets is honored and there are no disruptions at random times outside of schedule.

Reproduction Steps (Please include YAML):

  • Install Karpenter 1.0.8 to EKS cluster
  • Create EC2NodeClass and NodePool with schedule in disruption.budgets, examples:
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: some-pool
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 0s
    budgets:
      - schedule: 0 8 * * mon-fri # working days and hours
        duration: 8h
        nodes: 10%
        reasons: [Empty, Underutilized]
      - nodes: "0"
        reasons: [Drifted] # disable drift disruption

⏫ with this budget Empty/Underutilized are happening at any time instead of schedule: 0 8 * * mon-fri

---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: other-pool
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 0s
    budgets:
      - schedule: 0 8 * * mon-fri # working days and hours
        duration: 8h
        nodes: 10%

⏫ with this budget all disruptions, including Drift are happening at any time. For example, when new AL2023 AMI is released, all nodes are re-provisioned, ignoring schedule.

Other:

It was reported couple of times in other issues, but I wasn't able to find clear issue exactly about non-working schedules:

  1. AMI drift triggers node recreation outside Disruption schedule aws/karpenter-provider-aws#7592
  2. Pod disruption schedule #1719 (comment)
  3. Karpenter does not respect reasons in budgets #1691 (comment)

Versions:

  • Chart Version: 1.0.8
  • Kubernetes Version: v1.30.8-eks-2d5f260
  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment
@nantiferov nantiferov added the kind/bug Categorizes issue or PR as related to a bug. label Jan 28, 2025
@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Jan 28, 2025
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If Karpenter contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
None yet
Development

No branches or pull requests

2 participants