Scoping of "shared LLM"

[SigneA-hm]

The steering commitee wish to investigate the possibility to share ressources across organisations in order to cut costs for hosting LLMs.

First we need to investigate which market standard apply currently.
@janhalen, @lilosti og @cableman

Can you please provide me with an overview of marketstandard:

• Names of european hosting providers able to provide a large cluster for multiple organizations accessing the same LLM(s).
• Short description of their service offering concernng shared LLMs

@lilosti and @cableman you have a timebox og 5 hours to investigate this.

Does this task make sense to you? Please provide me with a timeframe for the task.

[janhalen]

The biggest challenge i see in a "vendor solution" approach is keeping your digital sovereignty and ownership to your AI solutions while making sure the solution operates 100% inside an EU jurisdiction.

If an Open Source "LLM as a Service" provider is found, how could a later move to another provider be guaranteed without securing and owning the deployments and documentation?

The most efficient and lowest cost (in the long run) way of doing this would be to settle on a "GPU" hosting provider and employ an experienced k8s cluster operations team to compose a "LLM as a Service" stack consisting of existing modern Cloud Native Components that aims to solve the concerns regarding "fair-sharing" - "pod scaling" - "ressource queues" and "access polices". Currently Im not aware of a single "big-bang" end-to-end system or supplier that offers a solution to all these concerns.

What I can do though is to describe a standardized loosely coupled reference architecture to be followed. That is if the steering comittee wishes to get the most out of the current kubernetes investment and solve this via modern standard cloud native tooling.

[janhalen]

@SigneA-hm: Are you 100% sure you want to hold the discussion about this topic in the documentation repository and not in the governance repository? Its pretty easy to move it, no worries...

[janhalen]

I have described a reference architecture that anchors credential control, security, endpoint abstraction and fair-user / scaling on the infrastructure and inference domain levels. (Outside the application domain), using only modern cloud native standard components.

Such an architecture frees up the responsibilities from the AI - application developer and lets them concentrate on delivering business value instead of building custom infrastructure inside the application domain.

Proposal here: https://janhalen.github.io/enterprise-architecture-patterns/proposals/20025-11-25-llm-as-a-service.html

[SigneA-hm]

@lilosti will you please add a timeframe for this issue?

[lilosti]

@SigneA-hm jeg har lagt den i næste sprint, det slutter 20/3, så det vil være færdigt inden da.

[DKAndreasen]

@SigneA-hm what do you mean by "... able to provide a large cluster for multiple organizations accessing the same LLM(s)"

• Do you want providers of large cluster, so that some operator, that os2 chooses, can deploy an infrastructure that will allow multiple organizations accessing the same LLM?... That is you are simply looking for providers of large k8s clusters, right (no matter the final usecase)? And then it is the infrastructure that the operator deploys that deals with the sharing among multiple orgs.
• Or are you looking for providers, where multiple organisations can sign up and have "llm-as-a-service"? (There's multiple of them, right??)

Of providers, I am aware of

• Telenor AI Factory: owns the hardware, that support at least some of bineric's model, who again directly provides inference (aka llm-as-a-service)
• Scaleway that offers managed k8s and according to their site can provide clusters up to 8 H100 and they also have new blackwell GPU hardware on the way
• Then there's also the planned colab betw Gefion and Netcompany, that you are aware of
• There's Ordbogen AI which might be more inference provider than hardware provider (and their current model offer is not open weights/source), but they boast of having a large blackwell cluster, so maybe they'll be interested in some kind of alternative deal, where they host a model of our choice on the side of their current offer.
• and as we have also discussed there are Alexandra Instittuttet, that want to be an inference provider as well

On the pure open weight european inference provider I think we still have mistral as well.

[DKAndreasen]

The biggest challenge i see in a "vendor solution" approach is keeping your digital sovereignty and ownership to your AI solutions while making sure the solution operates 100% inside an EU jurisdiction.

If an Open Source "LLM as a Service" provider is found, how could a later move to another provider be guaranteed without securing and owning the deployments and documentation?

@janhalen why is OpenResponse API standard not enough to ensure that we can move to any other provider of inference/LLM-as-a-service?

[janhalen]

@DKAndreasen: Great point and such an API would be a solid approach.

But I have yet to find any providers offering this.. yet alone more than one.

But yes I fully agree! A standardized API would be the first criteria to demand that a hosting provider implement and luckily there are several potential APIs.

This is excactly my point. We shouldnt "go shoppping" but instead define a solid architecture and select a supllier that can implement and operate the defined and accepted architecture. (And yes, again this architecture should include a support for the needs we have like inference, fair-use etc.)

[DKAndreasen]

But Scaleway provides that: managed inference or pay-per-token-inference or am I missing something?

and mistral provide it for both their open and closed weights models?

[janhalen]

These are good examples of inner-source solutions, that cannot be migrated without writing new code and new configurations. The deployment-charts are not obviously acessible, and that is perfectly understandable given their business model. But it still introduces a soft vendor lock, even though the are exposing APIs that are well documented.

But thats all just about technical and implementation risks..

What is the most important for the project is that they can actually deliver on the goals set by the steering comittee.

• Fair-use scaling
• Fixed price - calculated from the organisations size.

I'm not sure that they actually can, but given the needed ressources to invest in contacting these suppliers and explaining the needs, it might be possible to land a deal with Scaleway or Mistral?

[DKAndreasen]

These are good examples of inner-source solutions, that cannot be migrated without writing new code and new configurations.

On both of the mentioned platforms mistral-small-3.2-24b-instruct-2506 is available. The only thing I see necessary to switch betw those would be to change the api endpoint url and the token - that new config true, but very minor and no new code.

For sure then, models will retire, and that will force us to update prompts and maybe code or processes and that'll be a pain in the ass. But even if we were in charge and could decide to keep running ancient models, then new models would shape the imagination and automation possibilities and we would want to run those models too and at some point I am sure, we would want to streamline the number of models we have running (to reduce cost, if nothing else) and so we would want to retire models and update out legacy.

The deployment-charts are not obviously acessible, and that is perfectly understandable given their business model. But it still introduces a soft vendor lock, even though the are exposing APIs that are well documented.

Fully agree, but still also, choosing any operator for anything will provide the operator with experience and that will introduce very soft vendor locks...

What is the most important for the project is that they can actually deliver on the goals set by the steering comittee.

* Fair-use scaling

* Fixed price - calculated from the organisations size.

Could you please again explain this "fair-use scaling"? From your blog post you talk about GPU quota and controlling who can access the GPU at any given moment, but in order to run large language models efficiently you batch the requests and this is done dynamically, so an end-user is not waiting for their query to be answered in some queue (as long as the batch size is not maxed out). And as I understands this (but more knowledgeable people needs to check this), the larger the batch size (load on the system) the slower the model computation will be, thus the time-to-first-token and tokens-per-second will worsen. So yes in a period of high usage you need to scale up the inference service with another GPU (or GPU set if the model is distributed across). That would happen if batches are maxed out, so that you actually have queuing or if TTFT exceeds the threshold we agree on.

Anyway, my question being, how to define fair-use in that scenario? ... On what time-scales are we taking fair-use?

I'm not sure that they actually can, but given the needed ressources to invest in contacting these suppliers and explaining the needs, it might be possible to land a deal with Scaleway or Mistral?

Agree, I chatted with Rasmus from Alexandra Instituttet yesterday and they are also interested in talking about what we need and what they can provide, but there I also had trouble defining exactly what is meant by fair use, so I think we'll have to be more precise there. And I am sure it goes for all the other providers I listed above and for computerome/deranged, that they'll be up for discussing how they could help us.