[Workflow Pause] Proto changes #653

gow · 2025-10-22T21:58:55Z

Adding proto changes that are required to implement Workflow Pause/Unpause feature. Added,

WorkflowExecutionStatus.WORKFLOW_EXECUTION_STATUS_PAUSED enum
EVENT_TYPE_WORKFLOW_EXECUTION_PAUSED & EVENT_TYPE_WORKFLOW_EXECUTION_PAUSED event enums and corresponding event attributes
PauseWorkflowExecution & UnpauseWorkflowExecution gRPC along with corresponding request/responses.

Why?
These changes are needed to implement workflow pause/unpause feature.

Breaking changes
Introduction of new status WORKFLOW_EXECUTION_STATUS_PAUSED may break any saved visibility queries that may have assumed WorkflowExecutionStatus != 'Running' returns all closed workflows.

cretz · 2025-10-22T22:49:04Z

temporal/api/history/v1/message.proto

+message WorkflowExecutionPausedEventAttributes {
+    // The identity of the client who paused the workflow execution.
+    string identity = 1;
+    // The reason for pausing the workflow execution.
+    string reason = 2;
+    // The request ID of the request that paused the workflow execution.
+    string request_id = 3;
+}
+
+// Attributes for an event marking that a workflow execution was unpaused.
+message WorkflowExecutionUnpausedEventAttributes {


I think you forgot to add these to the actual oneof below

Yes. I had forgotten them. Added now.

I think we might want to wait to merge this until the implementation is present so we can confirm these types of things work with the code

Yes. I will not merge until the corresponding server PR using this is not ready.

cretz · 2025-10-22T22:49:48Z

temporal/api/workflowservice/v1/service.proto

+    // PauseWorkflowExecution pauses the workflow execution specified in the request.
+    rpc PauseWorkflowExecution (PauseWorkflowExecutionRequest) returns (PauseWorkflowExecutionResponse) {


We should clarify what pause actually does in docs here

Added more detailed description. Also annotated that these APIs are experimental

cretz · 2025-10-22T22:50:41Z

temporal/api/workflowservice/v1/request_response.proto

+    // The identity of the client who initiated this request.
+    string identity = 3;
+    // Reason to pause the workflow execution.
+    string reason = 4;


Can I see this reason in visibility? Or at least via describe?

Yes. The reason will be indexed in visibility and also be returned in describe response (to be shown in CLI & WebUI)

I don't see it in the protos that come back from list/describe. We aren't making a new search attribute for this right?

I had missed them. Will update.

Added WorkflowExecutionPauseInfo

Re: search attribute - I think we can reuse an existing attribute. Will check.

cretz

I saw this targets main. Can the implementation be completed before we do that?

cretz · 2025-10-24T12:43:56Z

temporal/api/workflowservice/v1/request_response.proto

+    // The identity of the client who initiated this request.
+    string identity = 3;
+    // Reason to pause the workflow execution.
+    string reason = 4;


I don't see it in the protos that come back from list/describe. We aren't making a new search attribute for this right?

cretz · 2025-10-24T12:44:29Z

temporal/api/history/v1/message.proto

+message WorkflowExecutionPausedEventAttributes {
+    // The identity of the client who paused the workflow execution.
+    string identity = 1;
+    // The reason for pausing the workflow execution.
+    string reason = 2;
+    // The request ID of the request that paused the workflow execution.
+    string request_id = 3;
+}
+
+// Attributes for an event marking that a workflow execution was unpaused.
+message WorkflowExecutionUnpausedEventAttributes {


I think we might want to wait to merge this until the implementation is present so we can confirm these types of things work with the code

cretz · 2025-10-24T12:45:33Z

temporal/api/workflowservice/v1/service.proto

        };
    }
+
+    // Note: This is an experimental API and the behavior may change in a future release.


Note, this doesn't mean we can just change things at will. People deploy proxies on certain API versions and may not upgrade them immediately on every backwards incompatible change. We have to try our best to not land it until we're pretty happy with the implementation.

cretz · 2025-10-24T12:46:55Z

temporal/api/workflowservice/v1/service.proto

+    // - The workflow execution status changes to `PAUSED` and a new WORKFLOW_EXECUTION_PAUSED event is added to the history
+    // - No new workflow tasks or activity tasks are dispatched.
+    //   - Any workflow task currently executing on the worker will be allowed to complete.
+    //   - Any activity task currently executing will be allowed to complete. But the heartbeat response will be annotated with 'activity_paused=true'.


Pedantic doc note - It won't be allowed to complete if the activity is properly getting interruptions. Also, this doesn't cover activities backing off. I'd rephrase this to just say activities currently running are paused and let the definition of activity pause be enough instead of trying to re-describe it here.

cretz · 2025-10-24T12:47:37Z

temporal/api/workflowservice/v1/service.proto

+    // - No new workflow tasks or activity tasks are dispatched.
+    //   - Any workflow task currently executing on the worker will be allowed to complete.
+    //   - Any activity task currently executing will be allowed to complete. But the heartbeat response will be annotated with 'activity_paused=true'.
+    // - Signals, child workflows, timers & timeouts, activity completion etc will continue to be processed by the server. No completed work is lost during pause.


Pedantic doc note - I would just say "all server-side events will continue to be emitted" and not try to list them here since they'll become stale.

cretz · 2025-10-24T12:49:07Z

temporal/api/workflowservice/v1/service.proto

+    // UnpauseWorkflowExecution unpauses a previously paused workflow execution specified in the request.
+    // Unpausing a workflow execution results in
+    // - The workflow execution status changes to `RUNNING` and a new WORKFLOW_EXECUTION_UNPAUSED event is added to the history
+    // - New workflow task and activity tasks are dispatched (if required).


Pedantic doc note - I think you should just say activities are unpaused and use existing activity unpause definition instead of attempting to re-describe here

gow · 2025-10-24T17:23:39Z

I saw this targets main. Can the implementation be completed before we do that?

Absolutely, I'll leave this up in this branch and not merge it until server PR is ready. Did you have a different workflow in mind?

cretz · 2025-10-24T18:36:29Z

Did you have a different workflow in mind?

Nope, this works for me. I only care about main heh.

spkane31 · 2025-10-29T15:58:22Z

temporal/api/history/v1/message.proto

    temporal.api.enums.v1.UpdateAdmittedEventOrigin origin = 2;
 }

+    // Attributes for an event marking that a workflow execution was paused.


Suggested change

// Attributes for an event marking that a workflow execution was paused.

// Attributes for an event marking that a workflow execution was paused.

bergundy

Left a few small non-blocking comments.

bergundy · 2025-10-30T01:04:46Z

temporal/api/workflowservice/v1/request_response.proto

+    // Namespace of the workflow to unpause.
+    string namespace = 1;
+    // Execution info of the workflow to unpause.
+    temporal.api.common.v1.WorkflowExecution workflow_execution = 2;


Debating whether we want to keep using this struct, we decided against a wrapper struct for this id, run_id tuple for all of the activity APIs.

Yeah. I think we decided to inline these fields. Will make the change.

bergundy · 2025-10-30T01:06:15Z

temporal/api/history/v1/message.proto

+    // The ID of the `WORKFLOW_EXECUTION_PAUSED` event that this unpause corresponds to.
+    int64 paused_event_id = 4;


Is this needed? It's going to be a bit of a hassle to implement due to buffered events but not a big deal, just beware.

I thought I may need it to quickly find the pause event when we are cherry picking the events. I'll remove it for now and add it if needed.
Re: buffered events - this should not be an issue. By the time we record unpause event, we are guaranteed to have the pause event recorded and the event ID generated for it.

Removed this for now. Did not need it.

bergundy · 2025-10-30T01:07:01Z

temporal/api/enums/v1/workflow.proto

    WORKFLOW_EXECUTION_STATUS_TERMINATED = 5;
    WORKFLOW_EXECUTION_STATUS_CONTINUED_AS_NEW = 6;
    WORKFLOW_EXECUTION_STATUS_TIMED_OUT = 7;
+    WORKFLOW_EXECUTION_STATUS_PAUSED = 8;


Verifying that we have consensus for adding this new status... this is fine with me FTR.

I think I am the only holdout. I still think it breaks Temporal expectation that a workflow is either running or closed, and that running has nothing to do with whether tasks are being processed (or signals or activities or timers). Having said that, I think unfortunately people are wanting to break these expectations.

I also think it's confusing to assume the entire workflow is in a paused status just because some things are paused. When a user asks us to support, say, pausing only signals/updates, it's gonna look real strange that we chose to treat pause as a status instead of what it truly is - a collection of paused aspects.

We had the consensus in the internal discussion about this. The reasoning was that pausing a workflow would set an expectation that the workflow status to change to PAUSED. Otherwise "workflow pause" operation appears broken.
If (and when) we support pausing of individual aspects we would have individual status for them to make it clear that only certain aspects are paused.

We had the consensus in the internal discussion about this

👍

The reasoning was that pausing a workflow would set an expectation that the workflow status to change to PAUSED

I don't think this is true that it would set this expectation any more than "cancel" would set an expectation of a workflow status changing to "cancel requested". Basically, there is no such thing as pausing a workflow, just pausing certain parts. I want to warn about user confusion by marking what Temporal deems to be a started/running workflow as not running.

If (and when) we support pausing of individual aspects we would have individual status for them to make it clear that only certain aspects are paused.

To clarify here, you mean the workflow status would remain running if we only paused signals, updates, etc? Or we would set it to paused even if some of it is paused? That's what we're doing with this change, e.g. the timers are not paused, but we are acting as if the whole workflow is paused when it is not.

I'm actually not completely sure. It depends on how we support these.
If pausing these individual aspects is a top level operation then it would have a separate status (like activity pause). If it is an option to workflow pause then the workflow status would be changed.
I'm also not sure if we will support pausing just the individual aspects without pausing the workflow tasks.

I'm also not sure if we will support pausing just the individual aspects without pausing the workflow tasks.

Note, you are making a permanent "can't take back" change here for something you may not be sure about in successive phases. But if we are defining status "paused" as "workflow tasks are paused" and not some general definition of "pause", ok (even if I do disagree with the premise).

…ntAttributes

This reverts commit 0f38a7a.

gow requested review from a team as code owners October 22, 2025 21:58

gow marked this pull request as draft October 22, 2025 21:59

gow marked this pull request as ready for review October 22, 2025 22:43

cretz reviewed Oct 22, 2025

View reviewed changes

gow requested a review from cretz October 23, 2025 18:54

cretz reviewed Oct 24, 2025

View reviewed changes

gow requested review from bergundy and spkane31 October 29, 2025 00:32

gow mentioned this pull request Oct 29, 2025

Mutable state changes for workflow pause. temporalio/temporal#8560

Open

5 tasks

spkane31 reviewed Oct 29, 2025

View reviewed changes

bergundy approved these changes Oct 30, 2025

View reviewed changes

gow force-pushed the cg/wf_pause_1 branch from 2fe3b23 to 0f38a7a Compare October 31, 2025 17:54

gow added 11 commits November 6, 2025 14:51

[Workflow Pause] Proto changes

7efd5cb

make proto

fbd9785

Add attributes to oneoff and add docs

b076f2a

make proto

e261a75

Added WorkflowExecutionPauseInfo and addressed comments

414ead2

make proto

1892424

Inline worklfow_id & run_id

6a5e637

Use Id instead of workflow_id

cef26c1

Removed reference to pause event ID from WorkflowExecutionUnpausedEve…

16af4ae

…ntAttributes

Revert "Use Id instead of workflow_id"

b2a5959

This reverts commit 0f38a7a.

make proto

5d8eed6

gow force-pushed the cg/wf_pause_1 branch from e6a5a42 to 5d8eed6 Compare November 6, 2025 22:52

		// PauseWorkflowExecution pauses the workflow execution specified in the request.
		rpc PauseWorkflowExecution (PauseWorkflowExecutionRequest) returns (PauseWorkflowExecutionResponse) {

	// Attributes for an event marking that a workflow execution was paused.
	// Attributes for an event marking that a workflow execution was paused.

		// The ID of the `WORKFLOW_EXECUTION_PAUSED` event that this unpause corresponds to.
		int64 paused_event_id = 4;

[Workflow Pause] Proto changes #653

Are you sure you want to change the base?

[Workflow Pause] Proto changes #653

Conversation

gow commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cretz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cretz Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gow commented Oct 24, 2025

Uh oh!

cretz commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bergundy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gow Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cretz Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cretz Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gow commented Oct 22, 2025 •

edited

Loading

cretz Oct 24, 2025 •

edited

Loading

cretz commented Oct 24, 2025 •

edited

Loading

gow Oct 31, 2025 •

edited

Loading

cretz Oct 30, 2025 •

edited

Loading

cretz Oct 31, 2025 •

edited

Loading

cretz Nov 7, 2025 •

edited

Loading