Support longer documents

[geoalgo]

Currently, the examples truncate all documents larger than the context-length supported by the model.

We would need to support inference on documents with any length (doing inference by batch and recombining) if we want to apply it to large corpus such as nemotron-cc.

[maxidl]

For now, our goal is that inference-hive can be seen as similar in spirit to the openai batch api. It does support any input length (up to context length set by user in the inference server config).

I think that what you suggest should not be part of inference-hive itself, but is rather up to the user to deal with it. For example, if you want to translate a document of 100k tokens, but the model only supports a 32k context, the user could split the document into chunks, get the responses for using inference-hive, and then combine the responses accordingly.

So while i dont think we will add specific features for this to inference-hive, I can create an example that demonstrates how this could be done in terms of preparing inputs and post-processing outputs.

[geoalgo]

Thanks, it makes sense if there is an example. Ultimately, what is important is to support this when we translate Nemontron-cc high for pretraining (or equivalent).