feat: Add TERM handling by fatmcgav · Pull Request #375 · jenkinsci/swarm-plugin

fatmcgav · 2021-10-13T21:46:02Z

This is useful to gracefully stop the Jenkins swarm process, for example
when running in Kubernetes in conjunction with a preStop lifecycle hook.

Signed-off-by: Gavin Williams gavin.williams@elastic.co

Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
Ensure that the pull request title represents the desired changelog entry
Please describe what you did
Link to relevant issues in GitHub or Jira
Link to relevant pull requests, esp. upstream and downstream changes
Ensure you have provided tests - that demonstrates feature works or fixes the issue

Example log output:

Oct 13, 2021 10:38:21 PM hudson.plugins.swarm.Client run
WARNING: Remaining retries: 5
Oct 13, 2021 10:38:21 PM hudson.plugins.swarm.Client run
INFO: Retrying in 10 seconds
Oct 13, 2021 10:38:30 PM hudson.plugins.swarm.Client$1 handle
INFO: Exiting...

basil

Thanks for yet another contribution, Gavin! At a high level, this makes sense to me. At an implementation level, I wonder if this could be done without resorting to the private sun.misc.Signal API, which should generally be avoided for forward compatibility with future Java versions. I did a quick Google search and found that the official Runtime.getRuntime().addShutdownHook API should be able to handle SIGTERM. Can you look into using that API rather than the private sun.misc.Signal API to implement this RFE?

fatmcgav · 2021-10-14T06:16:57Z

@basil thanks for the feedback...

I did start out with the addShutdownHook api but couldn't find a good implementation.. I'll try again though.. 👍

fatmcgav · 2021-10-18T10:01:52Z

@basil I've replaced the use of sun.misc.Signal with Runtime.getRuntime().addShutdownHook().

PTAL :)

This is useful to gracefully stop the Jenkins swarm process, for example when running in Kubernetes in conjunction with a preStop lifecycle hook. Signed-off-by: Gavin Williams <gavin.williams@elastic.co>

basil · 2021-10-19T18:56:31Z

client/src/main/java/hudson/plugins/swarm/Client.java

+                        logger.info("Interrupting '" + th.getClass() + "' termination");
+                        th.interrupt();


Have you verified, when sending a TERM signal to a connected Swarm client, that this if statement is true and that the code inside the if statement is executed?

Yeh, so I've confirmed that doing a kill <PID> results in the log line being output:

java -jar client/target/swarm-client.jar -description "fatmcgav swarm test" -disableClientsUniqueId -executors 1 -fsroot jenkins -retry 6 -maxRetryInterval 160 -retryBackOffStrategy exponential -mode exclusive -username local-swarm -password **** -deleteExistingClients -name fatmcgav-swarm-test -master https://*** -username local-swarm -webSocket -labels swarm -labels x86_64 Oct 19, 2021 9:51:54 PM hudson.plugins.swarm.Client logArguments INFO: Client invoked with: -deleteExistingClients true -description fatmcgav swarm test -disableClientsUniqueId true -executors 1 -fsroot jenkins -labels [swarm, x86_64] -maxRetryInterval 160 -mode exclusive -name fatmcgav-swarm-test -password ***** -retry 6 -retryBackOffStrategy EXPONENTIAL -url https://*** -username ***** -webSocket true Oct 19, 2021 9:51:54 PM hudson.plugins.swarm.Client run INFO: Connecting to Jenkins controller Oct 19, 2021 9:51:54 PM hudson.plugins.swarm.Client run INFO: Attempting to connect to https://*** ... Oct 19, 2021 9:52:05 PM hudson.plugins.swarm.Client run WARNING: Remaining retries: 4 Oct 19, 2021 9:52:05 PM hudson.plugins.swarm.Client run INFO: Retrying in 20 seconds Oct 19, 2021 9:52:05 PM hudson.plugins.swarm.Client$1 run INFO: Interrupting threads

Yeh, so I've confirmed that doing a kill <PID> results in the log line being output:

java -jar client/target/swarm-client.jar -description "fatmcgav swarm test" -disableClientsUniqueId -executors 1 -fsroot jenkins -retry 6 -maxRetryInterval 160 -retryBackOffStrategy exponential -mode exclusive -username local-swarm -password **** -deleteExistingClients -name fatmcgav-swarm-test -master https://*** -username local-swarm -webSocket -labels swarm -labels x86_64 Oct 19, 2021 9:51:54 PM hudson.plugins.swarm.Client logArguments INFO: Client invoked with: -deleteExistingClients true -description fatmcgav swarm test -disableClientsUniqueId true -executors 1 -fsroot jenkins -labels [swarm, x86_64] -maxRetryInterval 160 -mode exclusive -name fatmcgav-swarm-test -password ***** -retry 6 -retryBackOffStrategy EXPONENTIAL -url https://*** -username ***** -webSocket true Oct 19, 2021 9:51:54 PM hudson.plugins.swarm.Client run INFO: Connecting to Jenkins controller Oct 19, 2021 9:51:54 PM hudson.plugins.swarm.Client run INFO: Attempting to connect to https://*** ... Oct 19, 2021 9:52:05 PM hudson.plugins.swarm.Client run WARNING: Remaining retries: 4 Oct 19, 2021 9:52:05 PM hudson.plugins.swarm.Client run INFO: Retrying in 20 seconds Oct 19, 2021 9:52:05 PM hudson.plugins.swarm.Client$1 run INFO: Interrupting threads

Do you see a line above that says "Interrupting '[…]' termination"? I can't see one.

Good point... I'll do some debugging...

So I ran some tests locally, and even with the swarm-client successfully connecting to a Jenkins controller, I don't see any threads being interrupted.

I've put a couple of thread dump here: https://gist.github.com/fatmcgav/f848c071bbf5f650408c3e7f0b45238a

The first one is just with the agent idle and connected to Jenkins, the 2nd with a simple sleep script running...

It looks like the runningThreads set never gets populated with anything based on adding some additional logging...

So is it safe just to remove the logic? Or should the threads be getting interrupted?

basil

Based on https://github.com/jenkinsci/remoting/blob/f4b81bbb7b2df53bc14e1a14a36d4fd3ce570a31/src/main/java/hudson/remoting/jnlp/Main.java#L284-L287 it is sufficient to send an interrupt to the main thread, which will in turn interrupt the Engine thread. In Swarm's case, the main thread is the one running the while (true) loop in Client.java. (One way of reliably getting at this thread is to call Thread.currentThread() at the start of main() and hold on to the reference.) That thread should be sitting at line 235 of Client.java at idle, and when interrupted it would first hit the finally block of the snippet I linked to above (which calls engine.interrupt() to interrupt the Engine thread) then fall into the catch block at line 240 of Client.java, where we log a stack trace and eventually loop back around to reconnect. In your case you want to effect a clean shutdown, so you will want to change this to not log a stack trace and reconnect but rather to wait for the Engine thread to finish shutting down and then exit from the main thread cleanly without looping back around to reconnect. To distinguish this new case from the existing use cases you will likely need to add some sort of AtomicBoolean that is set by your new shutdown hook.

fatmcgav · 2021-10-21T06:40:19Z

@basil thanks for the great reply and the pointer...

I'll rework...

basil reviewed Oct 14, 2021

View reviewed changes

basil added the rfe label Oct 14, 2021

fatmcgav force-pushed the swarm_handle_shutdown branch from 167e22a to dc49946 Compare October 18, 2021 10:01

Add TERM handling

c5b5b1e

This is useful to gracefully stop the Jenkins swarm process, for example when running in Kubernetes in conjunction with a preStop lifecycle hook. Signed-off-by: Gavin Williams <gavin.williams@elastic.co>

fatmcgav force-pushed the swarm_handle_shutdown branch from dc49946 to c5b5b1e Compare October 18, 2021 10:24

basil reviewed Oct 19, 2021

View reviewed changes

basil reviewed Oct 20, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Comments

feat: Add TERM handling#375

feat: Add TERM handling#375
fatmcgav wants to merge 1 commit intojenkinsci:masterfrom
fatmcgav:swarm_handle_shutdown

fatmcgav commented Oct 13, 2021

Uh oh!

basil left a comment

Uh oh!

fatmcgav commented Oct 14, 2021

Uh oh!

fatmcgav commented Oct 18, 2021

Uh oh!

basil Oct 19, 2021

Uh oh!

fatmcgav Oct 19, 2021

Uh oh!

basil Oct 19, 2021

Uh oh!

fatmcgav Oct 20, 2021

Uh oh!

fatmcgav Oct 20, 2021

Uh oh!

basil left a comment

Uh oh!

fatmcgav commented Oct 21, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		logger.info("Interrupting '" + th.getClass() + "' termination");
		th.interrupt();

Uh oh!

Comments

Conversation

fatmcgav commented Oct 13, 2021

Uh oh!

basil left a comment

Choose a reason for hiding this comment

Uh oh!

fatmcgav commented Oct 14, 2021

Uh oh!

fatmcgav commented Oct 18, 2021

Uh oh!

basil Oct 19, 2021

Choose a reason for hiding this comment

Uh oh!

fatmcgav Oct 19, 2021

Choose a reason for hiding this comment

Uh oh!

basil Oct 19, 2021

Choose a reason for hiding this comment

Uh oh!

fatmcgav Oct 20, 2021

Choose a reason for hiding this comment

Uh oh!

fatmcgav Oct 20, 2021

Choose a reason for hiding this comment

Uh oh!

basil left a comment

Choose a reason for hiding this comment

Uh oh!

fatmcgav commented Oct 21, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants