Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support draining multiple node #469

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

support draining multiple node #469

Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
5cd6d5a
support draining multiple node
andrewhu-hcl Nov 24, 2021
4e8f556
Merge branch 'master' into support-multiple-node-drain
calebxu-hcl Jan 6, 2022
5811f05
Merge branch 'master' into support-multiple-node-drain
calebxu-hcl Jan 17, 2022
4abdd9c
node-drain: return error if no nodes specified
calebxu-hcl Jan 14, 2022
d449bc4
node-drain: ExperimentDetails.TargetNode is now Experiment.TargetNodes
calebxu-hcl Jan 14, 2022
05726d3
node-drain: run pre/post checks on all target nodes
calebxu-hcl Jan 17, 2022
96a94ec
node-drain: fix code formatting
calebxu-hcl Jan 19, 2022
96f5a4c
Merge branch 'master' into support-multiple-node-drain
uditgaurav Mar 8, 2022
319ea71
Merge branch 'master' into support-multiple-node-drain
uditgaurav Apr 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 42 additions & 29 deletions chaoslib/litmus/node-drain/lib/node-drain.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,37 +114,50 @@ func PrepareNodeDrain(experimentsDetails *experimentTypes.ExperimentDetails, cli
// drainNode drain the application node
func drainNode(experimentsDetails *experimentTypes.ExperimentDetails, clients clients.ClientSets, chaosDetails *types.ChaosDetails) error {

select {
case <-inject:
// stopping the chaos execution, if abort signal received
os.Exit(0)
default:
log.Infof("[Inject]: Draining the %v node", experimentsDetails.TargetNode)

command := exec.Command("kubectl", "drain", experimentsDetails.TargetNode, "--ignore-daemonsets", "--delete-local-data", "--force", "--timeout", strconv.Itoa(experimentsDetails.ChaosDuration)+"s")
var out, stderr bytes.Buffer
command.Stdout = &out
command.Stderr = &stderr
if err := command.Run(); err != nil {
log.Infof("Error String: %v", stderr.String())
return errors.Errorf("Unable to drain the %v node, err: %v", experimentsDetails.TargetNode, err)
}
targetNodes := strings.Split(experimentsDetails.TargetNode, ",")

common.SetTargets(experimentsDetails.TargetNode, "injected", "node", chaosDetails)
log.Infof("Target nodes list: %v", targetNodes)
for _, targetNode := range targetNodes {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also handle the panic case when the length of the list is zero and give an error message to provide the target node name.

Copy link
Member

@uditgaurav uditgaurav Jan 7, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we run chaos for more nodes we could perform the pre and post chaos check for all the target nodes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can also change experimentsDetails.TargetNode to experimentsDetails.TargetNodes in the structure and other places where it is used.


return retry.
Times(uint(experimentsDetails.Timeout / experimentsDetails.Delay)).
Wait(time.Duration(experimentsDetails.Delay) * time.Second).
Try(func(attempt uint) error {
nodeSpec, err := clients.KubeClient.CoreV1().Nodes().Get(experimentsDetails.TargetNode, v1.GetOptions{})
if err != nil {
return err
}
if !nodeSpec.Spec.Unschedulable {
return errors.Errorf("%v node is not in unschedulable state", experimentsDetails.TargetNode)
}
return nil
})
select {
case <-inject:
// stopping the chaos execution, if abort signal received
os.Exit(0)
default:
log.Infof("[Inject]: Draining the %v node", targetNode)

command := exec.Command("kubectl", "drain", targetNode, "--ignore-daemonsets", "--delete-emptydir-data", "--force", "--timeout", strconv.Itoa(experimentsDetails.ChaosDuration)+"s")
var out, stderr bytes.Buffer
command.Stdout = &out
command.Stderr = &stderr
if err := command.Run(); err != nil {
log.Infof("Error String: %v", stderr.String())
return errors.Errorf("Unable to drain the %v node, err: %v", experimentsDetails.TargetNode, err)
}

common.SetTargets(targetNode, "injected", "node", chaosDetails)

err = retry.
Times(uint(experimentsDetails.Timeout / experimentsDetails.Delay)).
Wait(time.Duration(experimentsDetails.Delay) * time.Second).
Try(func(attempt uint) error {
nodeSpec, err := clients.KubeClient.CoreV1().Nodes().Get(targetNode, v1.GetOptions{})
if err != nil {
if apierrors.IsNotFound(err) {
return nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we add a log here to specify that the resource was not found?

} else {
return err
}
}
if !nodeSpec.Spec.Unschedulable {
return errors.Errorf("%v node is not in unschedulable state", targetNode)
}
return nil
})
if err != nil {
return err
}
}
}
return nil
}
Expand Down