[Skill proposal] `serving-llms-on-instinct`

[iswaryaalex]

Proposed skill name

serving-llms-on-instinct

Does something like this already exist?

Yes — as documentation, a runbook, or internal guide

Where should this skill live?

Path B: authored in a product repo (HIP, ROCm, Ryzen AI, Lemonade, ...) and registered here

Catalog focus area

Cross-stack porting

Skill description

Description: Deploy and optimize LLM inference on AMD Instinct GPUs. Covers the full path from "I want to serve a model" to a running, benchmarked endpoint, including a DevCloud on-ramp for developers who don't have AMD hardware yet.

Flow: Trigger run -> Detect GPU ( if not found, trigger AMD Developer cloud setup) -> Decide VLLM vs SGLang Engine selection, and its Attention backends ( AITER, FA etc) ->Quark -> Env Vars -> Runtime

[danielholanda]

I like the overall scope, but I would recommend slightly decreasing it for phase 0 to make sure we can have a high-quality skill and grow from there.

[Mahdi-CV]

Thanks, I agree with narrowing the initial scope so Phase 0 can be high-quality and easy to validate.

A practical Phase 0 could focus on vLLM first, covering the models we already support through vLLM on AMD Instinct GPUs.

Proposed Phase 0 Scope

Build the initial skill around the following flow:

1. Discover the target environment

   • Detect the server configuration
   • Detect available GPU(s)
   • Identify GPU architecture and relevant runtime capabilities

2. Validate model fit

   • Check whether the requested model is supported
   • Confirm the model fits the available GPU memory and topology
   • Validate required precision, quantization, and backend support

3. Resolve the correct vLLM configuration

   • Pull the appropriate environment variables from the vLLM recipe
   • Select the required runtime parameters
   • Apply model-specific and GPU-specific settings

4. Launch the endpoint

   • Start the vLLM server
   • Return the endpoint URL
   • Return any required connection details or credentials

Later Phases

Phase 1

Add SGLang support once the vLLM path is stable and well-tested.

Phase 2

Add engine and backend selection logic to choose the most efficient serving path for a given model, GPU architecture, and deployment target.

That would keep Phase 0 focused on a complete, reliable serving path while still leaving a clear roadmap for SGLang support and later optimization work.