How to route execution to specific operators?

[ericlacher]

Hi Community

This is based on a discussion which started here.
I am going to set up a synapse kubernetes deployment, which will have multiple operators running on different node(pool)s (using the spec.selector feature as proposed by @cdavernas).
The expectation would be, that the spawned runners would then run on the very same node as the operator.

However, what I do not understand is, how can I route execution of a certain workflow to a specific operator?
It seems at this point, it happens completely randomly.

I have (local docker)

• an operator "Foo" in namespace "Foo"
• a default operator in namespace "default"
• a workflow in namespace "default"

Running this workflow multiple times in the UI causes the execution happening on a randomly picked operator. When I trigger it multiple times after another (without changing anything on the deployment), it will run on "Foo" as well as "default".

So, what's the envisioned way to specify an operator taking up a specific workflow?

Thanks,
Eric

PS: not sure this is related, but I can see the reported errors from here as well.

[JBBianchi]

Hello @ericlacher ,

I'm not entirely sure, but one thing you could try is adding a label when creating the WorkflowInstance resource, with the key operator and the value set to the name of the operator you'd like to target. Unfortunately, this feature doesn't exist in the UI currently, you'll have to make the call to the API manually.

#...
metadata:
  labels:
    operator: your-operator-name
#...

I haven't tested this setup myself, so I’d be curious to hear if it works for you. 🤞

[ericlacher]

@JBBianchi thanks for your input. I just tested this:

POST /api/v1/workflow-instances

{
  ...
  "metadata": {
    ...
    "labels": {
      "operator": "default-operator.default"
    }
  }
  ...
}

which did not work. I then inspected the created raw instance and tried to set a label when using the endpoint:

{
  ...
  "metadata": {
    ...
    "labels": {
      "synapse.io/operator": "default-operator.default"
    }
  }
  ...
}

also without success. I also tried different naming schemes for keys and values, but nothing worked.

[ericlacher]

...however, I now came across this code, and it indicates it should work as you described.

[ericlacher]

@cdavernas are these lines doing what they should?

• https://github.com/serverlessworkflow/synapse/blob/main/src/operator/Synapse.Operator/Services/WorkflowInstanceController.cs#L90
• https://github.com/serverlessworkflow/synapse/blob/main/src/operator/Synapse.Operator/Services/WorkflowInstanceController.cs#L115
• https://github.com/serverlessworkflow/synapse/blob/main/src/operator/Synapse.Operator/Services/WorkflowController.cs#L98
• https://github.com/serverlessworkflow/synapse/blob/main/src/operator/Synapse.Operator/Services/WorkflowController.cs#L123

E.g. on the claim method in WorkflowInstanceController we have

if (resource.Metadata.Labels != null && resource.Metadata.Labels.TryGetValue(SynapseDefaults.Resources.Labels.Operator, out var operatorQualifiedName) && operatorQualifiedName == this.Operator.Resource.GetQualifiedName()) return true;

Without knowing the details of this code, I would've assumed we want to avoid claiming, if there's a mismatch, so rather having

if (resource.Metadata.Labels != null && resource.Metadata.Labels.TryGetValue(SynapseDefaults.Resources.Labels.Operator, out var operatorQualifiedName) && operatorQualifiedName != this.Operator.Resource.GetQualifiedName()) return false;

(note != instead of == and true -> false)

[cdavernas]

@ericlacher Good intuition! The condition check is indeed invalid. It should instead:

1. Check that the operator label has been set.
2. If it has been set, it should check if it matches the current operator. If it does not, it should return false, it it does, it should return true, to avoid uselessly patching the resource.

[ericlacher]

@cdavernas I just created new images based on your latest changes. When I now run a workflow without any "synapse.io/operator" label, it seems it never gets claimed (or tagged with the operator label).
When using the label, everything works reliably.
In the UI, they show up being executed by some operator, but it does not show any value in the "operator" column in the workflow instances overview.

Also, setting a selector on the operator

spec:
  selector:
    realm: foo

then creating an instance using
POST /api/v1/workflow-instances

{
  ...
   metadata:
      labels:
         realm: foo
  ...
}

also will not set the label, so the operator column is not populated with any value.
So, I cannot validate what operator has actually been used for these cases.

[ericlacher]

FYI, I am set up with VS Code & .Net extension now, making myself familiar with it. Hopefully I will be more self sufficient in debugging, for the future.

Cool!!!

getting the operator running was a matter of minutes (although this was my first .Net project I set up locally) - you did a great job in setting sane defaults etc. making onboarding easy for new joiners ;)

[cdavernas]

You are right about mostly everything. However, the following statement is only partially correct:

as a direct result, I assume specifying a label (to match selector) when creating a WorkflowInstance is useless

Actually, if no selector has been set on the workflow, it could be set on instances to filter them out, but I'm aware that is an edge case.

So, as long as we do not introduce a way to seed Workflow (instead of WorkflowDefinition) this is conceptually impossible

Correct. The seeding feature was added as a convenience for a client, and was meant to be painless and straightforward. I now realize we should provide a more advanced approach by allowing, as you suggested, seeding Workflow resources.

Perhaps we could have seeding for Workflow metadata separately

Maybe. Would that be you preferred approach? @bvandewe WDYT?

When it comes to Workflow management, how are y'all managing the workflows, also considering CICD usage?

My clients are publishing their flows to a git repo, and are syncing them on Synapse volumes in their CICD pipeline.
I personally use a mix of that approach and the CLI.

[cdavernas]

getting the operator running was a matter of minutes (although this was my first .Net project I set up locally) - you did a great job in setting sane defaults etc. making onboarding easy for new joiners ;)

Excellent, happy you could get going quickly! Thanks for the great feedback ❤️

[ericlacher]

My clients are publishing their flows to a git repo, and are syncing them on Synapse volumes in their CICD pipeline.
I personally use a mix of that approach and the CLI.

Could you elaborate on this a bit more? Having a volume of workflows will not populate them with redis, afaik, so what logic is getting them into the DB from the volume?

[ericlacher]

Actually, if no selector has been set on the workflow, it could be set on instances to filter them out, but I'm aware that is an edge case.

Hmm, not sure I can follow, here.
If there is no label on the workflow, but a selector is present on the operator, the InstanceController will not be able to claim it, even if the instance has the label set, because the controller's Operator.Workflows has it filtered out, right?

[ericlacher]

also, coming back to the namespaces: what is their purpose, given they're ignored for workflow routing? Just filtering resources in the UI? Would it be desirable to apply them more strictly, e.g. also constraining workflow execution to operators in the same NS?