Killing Rez tasks in a render farm setup

[johhnry]

Hi,

We have a render farm setup where we launch renders in a Rez environment like: rez env houdini -- hrender ...

The issue we are facing is that when trying to kill a running task on a computer, the visible PID is Rez's one and not hrender's one for example.

I would have thought that killing Rez was killing the subprocess that was spawned but this is not the case and it stills run in the background. This is quite problematic since we launch renders on machines that are not only dedicated to the render farm and users must be able to kill jobs.

My question is: is there a reliable way to kill the child process in a Rez resolved environment?

I noticed that there is the rez env --new-session argument but I don't quite understand how it works on Windows and how it might solve our issue (I tried rez env --new-session maya -- maya and closed the terminal but nothing happened)

Thanks!

[christian-korneck]

what OS (Linux, Win, …) do you use?

[johhnry]

It's mainly Windows computers for now

[nerdvegas]

Ok yeah interesting, I can replicate this with a simple repro on linux. I thought we'd sorted this ages ago actually, the topic has definitely come up before.

…

On Fri, Mar 18, 2022 at 2:06 AM Christian Korneck ***@***.***> wrote: what OS (Linux, Win, …) do you use? — Reply to this email directly, view it on GitHub <#1250 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMOUSTNCL2S45FG2SAFXMLVANC6LANCNFSM5Q63BMZQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[JeanChristopheMorinPerso]

This was a big problem at RodeoFX (and I'm sure it's still is). This code is supposed to fix that problem if I remember correctly but in some cases it doesn't.

At the end of the day, such issues are always process group related.

[johhnry]

Hi @JeanChristopheMorinPerso,

Thanks for the answer. If I understand correctly Rez spawn all subprocesses in the same process group so when it catches a kill signal it also kills all the processes in that group?

Also it seems to only work on Linux isn't it?

[johhnry]

I also found a SO thread about killing the whole process tree on Windows: https://stackoverflow.com/questions/1230669/subprocess-deleting-child-processes-in-windows

Some answers seems too coupled with the windows api but this answer using psutil looks interesting...

[christian-korneck]

yep, that’s what I wanted to answer in case of Windows. Kill the process tree (taskkill /pid <pid> /f /t).

However. Some processes might spawn away (parent process not waiting, leaving processes with no parent process, so not part of the process tree). Some of this can be prevented by using tricks like start /wait cmd.exe /c <actual cmdline> as the cmdline to start a process (-> package definition). Some other stuff needs special handling.

[johhnry]

@christian-korneck The taskkill seems to kill the process tree by targeting a rez process, we'll investigate on that with Tractor. Thanks!

[nerdvegas]

Actually I was wrong about my repro, I was doing a SIGKILL not SIGTERM. SIGTERM works as expected, child procs are killed. SIGKILL leaves child procs hanging, but you're not gonna get any guarantees with it so that's acceptable. How specifically are you killing the rez proc?

…

On Fri, Mar 18, 2022 at 8:37 AM Jean-Christophe Morin < ***@***.***> wrote: This was a big problem at RodeoFX (and I'm sure it's still is). This code <https://github.com/nerdvegas/rez/blob/6bd2f451e2ca464abcdcca3c7060a81fca1c2765/src/rez/cli/_util.py#L206-L243> is supposed to fix that problem if I remember correctly but in some cases it doesn't. At the end of the day, such issues are always process group related. — Reply to this email directly, view it on GitHub <#1250 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMOUSVF3UDKV6FOPVIUQPDVAOQ2RANCNFSM5Q63BMZQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[johhnry]

We are using Pixar's Tractor render farm and from what I've read the service running on every machine send those signals in order: SIGINT, SIGTERM, and SIGKILL.

I also found the following post on the Pixar forum:

There are cases in which the process launched by tractor-blade will spawn subprocesses that are no longer part of the same "process group" -- sometimes to deliberately avoid being killed. Tractor-blade sends the relevant signal to the entire process group rooted in the process that it directly spawned. Sometimes creating a subprocess in a different group is unintended; for example backgrounding a process in a csh script will do it, whereas the same operation in a bash script keeps the subprocess within the same group.

Also for more details there's this --no-sigint option on blades described here (we don't currently use that):

You are likely to be just fine using --no-sigint everywhere. Most commercial apps or os tools that need to do some kind of clean up will handle both SIGINT and SIGTERM gracefully. Apps that don't care about cleanup will just be killed either way.

In general, SIGINT is a "softer" interrupt to send to a process because many simple scripts and user-written programs have clean-up handlers for ctrl-c types of interrupts, while they may not make the effort to handle SIGTERM. Ironically, that's why we send it first, because several python programers wanted to use simple try/except handlers for KeyboardInterrupt, rather than dealing with SIGTERM signal handlers.

[johhnry]

Also there's a post (from 2015) clearly explaining the issue we are currently experiencing:

Yes, I am guessing that the SIGINT is causing some parent/wrapper to be killed but the signal is either not delivered to the child processes, or is ignored by them.

Since the wrapper is the only one that tractor-blade knows about, and it did exit, then the blade believes it has successfully killed the script it launched. As a result, the blade does not escalate to trying SIGTERM or SIGKILL on the process group as well.

It may be better in your case to have the blade skip the SIGINT step, and start the kill procedure with a SIGTERM to the process group. You can do this by starting the tractor-blade with the "--no-sigint" command-line option.

[nerdvegas]

This sounds like something we should add a per-shell test for. This _should_ be working (rez should terminate child procs on a SIGINT as per https://github.com/nerdvegas/rez/blob/master/src/rez/cli/_util.py#L242) but perhaps there's an issue on windows?

…

On Fri, Mar 18, 2022 at 9:59 AM Christian Korneck ***@***.***> wrote: yep, that’s what I wanted to answer in case of Windows. Kill the process tree (taskkill /pid <pid> /f /t). However. Some processes might spawn away (parent process not waiting, leaving processes with no parent process). Some of this can be prevented by using tricks like start /wait cmd.exe /c <actual cmdline> as the cmdline to start a process (-> package definition). Some other stuff needs special handling. — Reply to this email directly, view it on GitHub <#1250 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMOUSQYFX3MFSY65TMBXCTVAO2NXANCNFSM5Q63BMZQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[nerdvegas]

created #1251

[johhnry]

Thanks for opening an issue 👍

[johhnry]

For anyone having the same issue, I solved it by changing the Tractor blade code for killing the process:

# TrSubprocess.py

def send_signal(self, sig):
  if subprocess.mswindows:
    subprocess.call(['taskkill', '/F', '/T', '/PID', str(self.pid)])

Now it uses Powershell to kill the process tree.