Skip to content
This repository was archived by the owner on Jul 22, 2024. It is now read-only.
This repository was archived by the owner on Jul 22, 2024. It is now read-only.

draining stuck on hard instance failure #8

@svenwltr

Description

@svenwltr

The node-drainer gets stuck on a hard instance failure. In this case:

  • the node gets NotReady in Kubernetes
  • the pods on this node get either Unknown or NodeLost
  • the instance is in Terminating:Wait state

It gets stuck, because:

  1. The node gets not removed from Kubernetes, because it is still in the ASG.
  2. The instance does not get removed from the ASG, because it waits for the draining.
  3. The draining cannot happen, because the node is actually gone.

Also the node is nor reachable via SSH nor ping.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions