Add execution frequency or delay when executing workflow in batches

[Semonxue]

Self Checks

• This is only for bug report, if you would like to ask a quesion, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• Pleas do not modify this template :) and fill in all the required fields.

Dify version

0.6.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

When run workflow in batches, sometime the LLM query will reach the frequency usage if I put multi LLM node in workflow.
The idea is if can add a config for batch run and setting up the delay in every run.

✔️ Expected Behavior

The workflow run can support sequence or parallel mode , and support delay if the LLM api is lower query frequency.

❌ Actual Behavior

Currently , the run mode will reach limitation of query frequency on some LLM , like 01Yi, etc.

[Selene29]

To add to this:

Generally i would want to set a rate limit per api call.
(globally and/or locally)

I played around with embedding model from Cohere, where i would give it a longer file.

Cohere allows you 100 api calls per minute with a trial key.

In this case the embedding slammed the api, till it hit the limit and then lost all progress in the process.

And no, this is not a case of pay for more.
I dont need the results instantly, i can wait a while for the task to finish.
Even a delay of 1 second would already solve the problem.

[FarVision2]

As an aside, embeddings take almost no processing power. I use nomic-embed-text as well as mxbai-embed-large, with Ollama. The embeds pass through practically faster than you can read the logs. It is much wiser to spend your API calls on generation and tooling.

[Selene29]

Yeah, no contest from me.
Except Cohere is also free as long as you dont hammer their api.
That is the main point.

And currently i cant control this.

[Semonxue]

A rate limit is a good idea. but if it works on batch will be better. I mean how many jobs run at same time in batch mode.

[StreamlinedStartup]

Many api calls have rate limits and the Dify workflow tool, when running in batch mode, is just way too fast. I couldn't find any settings anywhere to limit this.

[StreamlinedStartup]

Adding a +1 here, this is absolutely necessary to prevent overloading APIs.