We have fine-tuned an open-weight model (Ministral) inside Azure AI Foundry (Azure ML). Our end goal is to run this fine-tuned model on local/mobile devices using Foundry Local.
Problem / Pain Point :
- Fine-tuning jobs in Azure AI Foundry produce standard PyTorch/Safetensors weights or LoRA adapter files.
- Foundry Local strictly requires models compiled into the ONNX runtime format (.onnx files with inference_model.json).
- Existing documentation primarily focuses on compiling base models directly from Hugging Face rather than exporting custom fine-tuned weights directly from Azure ML job artifacts to Foundry Local.
Questions:
Has anyone successfully set up or documented an end-to-end pipeline for this scenario? Specifically looking for guidance or recommended best practices on:
- Artifact Extraction: Recommended steps/SDK calls to pull fine-tuned model artifacts (.safetensors / adapter files) out of Azure AI Foundry / Azure ML job storage.
- ONNX Compilation: Standardized configs or scripts (e.g., using Microsoft Olive or onnxruntime-genai) to fuse LoRA adapters, convert, and quantize (INT4/FP16) locally downloaded weights into the ONNX format required by Foundry Local.
- Automated Tooling: Is there an existing or planned feature within Azure AI Foundry / Foundry Local to automate the ONNX compilation phase directly upon completion of a fine-tuning job?
Environment Details
- Base Model: Ministral
- Source: Azure AI Foundry / Azure ML Studio
- Target Runtime: Foundry Local (Mobile / Edge devices)
- Target Precision: INT4 ONNX
We have fine-tuned an open-weight model (Ministral) inside Azure AI Foundry (Azure ML). Our end goal is to run this fine-tuned model on local/mobile devices using Foundry Local.
Problem / Pain Point :
Questions:
Has anyone successfully set up or documented an end-to-end pipeline for this scenario? Specifically looking for guidance or recommended best practices on:
Environment Details