Local RAG Application with Foundry Local

A retrieval-augmented generation (RAG) application that answers questions about a collection of documents entirely on-device — no cloud, no API keys.

Based on the official Microsoft tutorial: Tutorial: Build a RAG application — Foundry Local

How it works

1. Knowledge base — a list of text documents defined in main.py.
2. Embeddings — each document is converted into a numerical vector using the local qwen3-embedding-0.6b model.
3. Retrieval — when you ask a question, it is embedded too, and the top-2 most similar documents are found using cosine similarity.
4. Generation — the retrieved documents are placed in the system prompt, and the local qwen2.5-0.5b chat model generates an answer grounded in that context (streamed token by token).

Question ──▶ embed query ──▶ cosine similarity vs. document vectors
                                        │
                                        ▼
                          top-2 relevant documents (context)
                                        │
                                        ▼
                    chat model ──▶ grounded, streamed answer

Prerequisites

• Windows, macOS, or Linux with at least 8 GB RAM
• Python 3.11+

Setup

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt

On macOS/Linux use pip install foundry-local-sdk openai instead of the Windows (-winml) package.

Run

python main.py

On first run, the two models are downloaded automatically and cached locally, so later runs start much faster. Then ask questions like:

• "What programming languages does the SDK support?"
• "How does Foundry Local run models?"
• "What is retrieval-augmented generation?"

Type quit to exit.

Files

File	Purpose
main.py	The complete RAG application
requirements.txt	Python dependencies