Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Re-order executions in waiting queue #324

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

RFC: Re-order executions in waiting queue #324

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a395afb
adding a file reorder-waiting-executions
Nov 26, 2022
b9beab7
Adding content
Nov 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 116 additions & 0 deletions rfc/reorder-waiting-executions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
# _Re-ordering waiting executions of a Pipeline_

_**IMPORTANT**: Fill in and remove all italicized text before submission.
If a section is not applicable, remove it._

| | |
|-|-|
| **Status** | _**Proposed**, Accepted, Implemented, Obsolete_ |
| **RFC #** | _Update after PR has been made_ |
| **Author(s)** | _Jalander Ramagiri (`@rjalander`)_
| **SIG / WG** | _Applicable SIG(s) or Working Group(s)_ |


## Overview

_Currently the waiting executions of a pipeline runs in FIFO order. But due to certain project urgencies and release purpose, we quite often need to change the executions order of a pipeline._

_Implementing the Re-order functionality helps to run the waiting executions in the order the user wants_

### Goals and Non-Goals

Goals:

_Introduce an option through Spin CLI and Spinnaker dashboard(GUI) to move the specific waiting execution of a Pipeline towards UP/DOWN or Top/Bottom from the current position, which is set in FIFO order by default_

_What problems are not not trying to solve?_

## Motivation and Rationale

_Currently it is not possible If the user wants to prioritize specific execution of a pipeline. Adding Re-order feature from CLI/GUI will help end-users and operators to change the priority order of a waiting execution for release purpose or any project urgencies._


## Timeline
_The code for CLI command reorder implementation is almost ready for review.
Planning to implement this feature in two Phases._

_Phase-1: completing CLI command implementation and make this feature available for customers to use and work on feedback if any._

_Phase-2: Implementing GUI for reorder functionality by using the backend that is available from CLI reorder implementation in Phase-1._

_Proposed timeline for Phase-1 would be 1-2 weeks as this feature is demoed for some Spinnaker users and got the feedback.
Phase-2 would be 3-4 weeks as there are some existing issues in the waiting executions area of Spinnaker dashboard that needs to be fixed._

## Design

_The Re-ordering waiting executions of a pipeline can be done through Spin CLI command and Spin dashboard._

_Introducing a new CLI command for reorder functionality of a waiting executions, the command will be used to Re-order waiting execution to UP/DOWN or Top/Bottom based on the current position of the provided execution._

_Example Spin CLI Command_
>>spin pipeline execution reorder --execution-id 01GJQ601MC3SBACVNZB0XC5GHJ --reorder-action UP

_New buttons will be added in Spinnaker dashboard(GUI) for each waiting execution to move the specific execution towards UP/Down or Top/Bottom based on the current position of the execution._

_New Gate API endpoint will be created for external access_
>>:8084/pipelines/<execution_id>/reorder?reorderAction=<UP/Down/Top/Bottom>

_New Orca API endpoint will be created to access the same way_
>>:8083/pipelines/<execution_id>/reorder?reorderAction=<UP/Down/Top/Bottom>

_The main business logic of the feature lies in Spin Orca service, where the **CompoundExecutionOperator** will update the Redis repo using the existing class **RedisExecutionRepository** and update the pending executions to the user selected position in the Redis repo using the Class **RedisPendingExecutionService**._
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note, would need to handle SQL as well if this moved forward.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for reviewing this, Jason. Do we need this to be implemented for DualPendingExecutionService and InMemoryPendingExecutionService too.


_A new handler class **ReorderWaitingExecutionHandler** will be created in orca-queue handler package, to handle the reorder of a waiting execution._

**_Note_**

_The Re-ordering will not be done for the executions which are not in Waiting state, an appropriate error message should be displayed._

_In the Spinnaker dashboard(GUI), only the waiting executions should have buttons to reorder, buttons should be removed once the execution is started._


### Dependencies

_This proposal depends on existing services Spin CLI, Spin Deck, Spin Gate and Spin Orca._

_No new dependencies are needed, existing systems will be sufficient to implement this feature_

## Drawbacks

_Currently there is no Re-order functionality of an execution_
_Why should we **not** do this?_
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some things that OFTEN hit:

  • How do you know what the order SHOULD be - by doing it in FIFO you handle SHA's which may not have a particular order, but you REALLY want the latest kinda thing.
  • E.g. Git triggers - it's possible that re-ordering would cause incorrect sha pulls on artifacts. SO there IS a question in my mind of whether this should by default be visible? It seems... rather specific to a very specific set of cases. Docker triggers would be similar where unintentionally you could deploy older docker images.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pipeline design will have significant bearing on usage of triggers in the delivery workflow. If the Git commit id from the trigger is used in the pipeline and not head of the branch and uses the state of pipeline context then it is possible to have two pipeline executions deliver correctly even if order is changed. However, it makes sense that one would want the latest.

There are couple of possibilities for reordering executions of the same pipeline

  • The pipeline is used by triggers from multiple branches and build from main branch needs to be prioritized in the queue while others need to execute as well
  • The pipeline is triggered by commits in same branch and multiple commits in the queue while the latest needs to be prioritized while others can be discarded

The first should use a different pipeline design and the second one should discard older executions in the queue as execution in different order can cause problems with identifying latest. This could then be an option of one execution at a time while discarding older execution in queue that has not yet started.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi,
Our Spinnaker users have similar request to re-order the waiting executions. But I think one of the most important reason is the FIFO is not working as expected in certain situations. I used to submit a ticket for this issue spinnaker/spinnaker#6373.

Can we consider to fix it first?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue with FIFO is not working is fixed with this PR spinnaker/orca#4356 and pending for review.

There are some users still they need to re-order the waiting executions even the FIFO is working as expected, for some release purpose or pipeline urgencies.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rjalander . Then it would be nice to also include this re-order feature . Our users will be happy to have it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One another scenario where Spinnaker users want to Re-order the waiting execution of same pipelines is,

  • When the Pipeline has waiting executions from different source of triggers and If we want to prioritize the specific source of trigger to validate on priority for any business needs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DEFINITELY want the FIFO fix in first, then... more discussion on impacts/changes required for this, which aren't in my mind exactly "small" at the moment. And this is MOSTLY tied to trigger impacts on executions and what data is pulled/used for triggers...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The FIFO defect is fixed

And we have done initial changes to bring this feature in and demoed to our internal Spinnaker users.

The changes are in this private branch - spinnaker/orca@master...rjalander:orca:spin-issue-6735, to reorder the executions with RedisPendingExecutionService.kt implementation.

And the CLI changes are in private branch - spinnaker/spin@master...rjalander:spin:spin-issue-6735-reorder, to initiate the reorder using CLI command.

Can we discuss more on impacts of this feature, to get this going.


## Prior Art and Alternatives
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you looked at the orca-qos API? There may be a way to extend that to implement this functionality.


_There is an alternate method that can be followed by cancelling all the running pipelines and Re-run the executions one after another in the right order in which the user wants. But this is a tedious process to run for every time to change the order of executions._

_This feature will help the user to re-order the executions any number of times until they are in waiting state._


## Known Unknowns

**Knowns**

_Will be implementing re-order feature using CLI/GUI for a waiting executions of a pipeline_

_Re-order with CLI will be implemented and released first to stabilize the features_

**Unknowns**

_The current implementation uses RedisPendingExecutionService and need to check on the requirement of implementing different pending execution services like DualPendingExecutionService, InMemoryPendingExecutionService and SqlPendingExecutionService._

## Security, Privacy, and Compliance

_As this feature is plugin to the existing component, need to check the security aspects with Security SIG or Technical Oversight Committee._

## Operations

_A new CLI will be created to re-order the waiting executions, that user can run using Spin CLI_
_And waiting executions in GUI will have an additional option/button to re-order the executions_

## Risks

_What happens If this feature not implemented with other pending executions services like DualPendingExecutionService, InMemoryPendingExecutionService and SqlPendingExecutionService._

## Future Possibilities