RFE: Cancel running Koji tasks on process restart #79

ELNBuildSync saves all of its active tasks in a database table, so if it crashes or is restarted for any reason (including process upgrade), it will start any currently-running batch over from the beginning. It can do this reasonably safely because it performs Koji draft builds and only promotes them to stable immediately prior to submitting them as a Bodhi update.

However, this means that any tasks that were running prior to the restart will keep going in Koji, tying up a builder and wasting resources on a build that will never be promoted.

Therefore, I think it is probably sensible that the startup process of EBS (before it begins processing any pending requests) should cancel any running Koji tasks that were started by the EBS user.

The workflow would look something like:

• EBS starts up and starts the logger and initiates its mainloop
• EBS downloads its dynamic configuration
• EBS gets the current user ID with session.getLoggedInUser()
• EBS looks up all active tasks for this user:

session.listTasks(
    opts={
        "owner": ebs_user_id,
        "state": [
            koji.TASK_STATES["FREE"],
            koji.TASK_STATES["OPEN"],
            koji.TASK_STATES["ASSIGNED"]
        ]
    }
)

• EBS cancels all remaining tasks owned by this user. Using the multicall() interface, cancel all at once.
• ELN finishes setting up and starts handling requests.

If the cancel attempts fail due to network issues, they should be retried, but if we get any other failure, we will just give up.