Enabling Upstage's Solar-Open-100B

[gilliean]

Hardware:

• 4x B60 with Xeon CPU

Model

• Upstage's Solar-Open-100B

As we try enable/run the model, it asks to use image upstage/vllm-solar-open:latest or install from source on https://github.com/UpstageAI/vllm.git@v0.12.0-solar-open. Seeing from its branch name v0.12.0, its vllm repo was updated on top of v0.12.0.

The model is one of the three finalists of Korea national AI models, and we'd like to enable the model proactively on B60/B70. Wanted to let the team know that XPU llm-scaler image with vLLM v0.12.0 is very much anticipated, and hopefully the Upstage's model is supported on it without too much effort.

[Wesley-Du]

Hi Gilliean, we are releasing llm-scaler-vllm b7 which is based on vllm 0.11.0, we may look into this model request when we start to work on 0.12.0 or higher.