[Bug]: During local embedding，RAGFlow is sending too much text at once, exceeding the model's maximum token limit, causing the model to be unable to fully read the input.

[XTFG]

Is there an existing issue for the same bug?

• I have checked the existing issues.

RAGFlow workspace code commit ID

main

RAGFlow image version

v0.15.1,nightly

Other environment information

Actual behavior

When employing embedding models with a lower 'maximum input token' capacity，models such as bge-large and conan-embedding-v1 are limited to a maximum input of 512 tokens. When using these models for embedding, RAGFlow sends more than 512 tokens at once, ollama will encounter an error.
I've found the cause of the error here：https://github.com/ollama/ollama/issues/7288#issuecomment-2591709109
Although I can adjust the maximum input limit of the model in ollama, it will cause RAGFlow's text to be truncated, resulting in incomplete embeddings.Additionally, I'm unable to locate a setting within RAGFlow to control the maximum input for the embedding model.

When adding a model, the max token setting controls the maximum output, not the input, which doesn't apply to embedding models.

The same issue of an ineffective max token option also exists when adding reranker models.

Expected behavior

Please add a setting to RAGFlow to control the maximum number of tokens sent to the embedding model per request, and also fix the bug where the max token limit is ineffective when adding reranker models.

Steps to reproduce

Using the bge-large:latest model in ollama, if the embedding is performed with a method other than 'general' (I am using 'book'), and the token count goes over 512, an error occurs and the embedding is terminated.

Additional information

No response

[KevinHuSh]

What about reducing chunk token size in chunking method settings?

But it will not slice text apart from the middle of text to ruin the semantics, which is meaningless to embedding.

[XTFG]

在分块方法设置中减小块令牌大小怎么样？

但它不会将文本从文本中间切开以破坏语义，这对嵌入毫无意义。

You know, some parsing methods, like QA, resume, manual, table, paper, laws, book, presentation, one, cannot manually set the Chunk token number.

[KevinHuSh]

We did not depends embedding a lot since its limitation of long text semantics representation.

[XTFG]

Could you please tell me what the significance of the 'max output tokens' setting is for embedding and rerank models?

[KevinHuSh]

Sometimes, if the input is too long, the serving of embedding reports error directly without automatically truncating.

[tusik]

Sometimes, if the input is too long, the serving of embedding reports error directly without automatically truncating.

Is that possible add OpenAI-API-Compatible configuration options, such as input length, I often encounter an error that exceeds 8196 when using paper mode for word embedding。