Skip to content

README.md

Latest commit

 

History

History
189 lines (137 loc) · 5.43 KB

README.md

File metadata and controls

189 lines (137 loc) · 5.43 KB
Raw

LokumAI — Local AI Chat Studio

A commercial-grade, local-first AI desktop app for macOS.
PyQt UI • MLX inference • Persistent RAG • Optional LoRA fine-tuning

Quickstart License Platform Local First

Overview

LokumAI is a desktop-first AI chat studio that runs fully on your machine:

  • A polished chat UI with streaming responses
  • RAG (Retrieval-Augmented Generation) over your local files with persistent storage
  • Optional MLX LoRA fine-tuning with a safe dataset pipeline (including ChatML-aware presplitting)

Designed for developers who want a fast local workflow without sacrificing reliability, persistence, or UX.

Features

🧠 Chat UX

  • Streaming tokens + Stop button
  • Chat history persisted in SQLite
  • Dev Mode tools embedded as a right sidebar (no separate window)

📚 RAG (Persistent Knowledge)

  • Index local folders (code + docs) and retrieve relevant context at chat time
  • Vector search with FAISS + sentence-transformers
  • Store survives restarts and is designed to be commit-safe

Supported inputs:

  • Text/code: .py .js .ts .md .txt .json .yaml ...
  • Documents: .pdf (PyMuPDF), .docx (python-docx)
  • Archives: .zim (libzim / python-zim)
  • Images (optional): OCR via pillow + pytesseract + tesseract

🧩 Fine-tuning (MLX LoRA)

  • One-click training runner via python -m mlx_lm lora ...
  • Live logs + graceful stop
  • Dataset helpers: JSONL (ChatML) + SQLite

Quickstart

1) Create + activate a virtualenv

python3 -m venv .venv
source .venv/bin/activate

2) Install dependencies

pip install -r requirements.txt

Optional OCR dependency (images):

brew install tesseract

3) Run

python3 -u main.py

Wait until the UI shows: Service: ready.

Configuration

prompts.json

prompts.json controls:

  • system prompt / user prompt
  • theme
  • model_path (MLX model directory)
  • use_rag (enable/disable RAG)

Environment variables (advanced)

Storage (local-first)

  • LOKUMAI_HOME — base app data folder (default: ~/.lokumai)
  • LOKUMAI_RAG_DIR — RAG store directory (default: ~/.lokumai/rag)
  • LOKUMAI_LORA_DIR — LoRA artifacts directory (default: ~/.lokumai/lora_data)
  • LOKUMAI_CHAT_DB — chat history DB path (default: ~/.lokumai/app.db)

Dev Mode password (no leaks)

  • LOKUMAI_DEV_PASSWORD — set your own password
  • otherwise the app stores/uses ~/.lokumai/dev_password.txt

Fine-tune memory shaping

  • LOKUMAI_FT_PRESPLIT=1 — enable presplitting (recommended)
  • LOKUMAI_FT_PRESPLIT_CHARS_PER_TOKEN (default 4.0) — lower = more aggressive split
  • LOKUMAI_FT_CLEAR_CACHE_THRESHOLD — lower = more frequent cache clears

RAG: persistent knowledge

RAG stores a cumulative index under your configured RAG directory.

Typical files:

  • faiss_index.bin — vector index
  • docs_metadata.npy — aligned chunk texts
  • chunks_meta.npy — per-chunk metadata
  • rag_state.json — per-file indexing state
  • rag_meta.json — convenience metadata

Reliability note:

  • If loading fails, LokumAI quarantines the store files (renames with .corrupt.<timestamp>) instead of silently appearing empty.

LoRA fine-tuning

Recommended baseline for large models (e.g. 27B 6-bit on Apple Silicon):

  • batch_size = 1
  • max_seq_len = 384 (then try 512)
  • run validation after training (training-time eval can spike memory)

Why training can OOM even with “free RAM”: Apple Metal memory can fail due to peak allocations + fragmentation, even if system monitors show headroom.

Dataset generation

Build a multi-turn dataset from prompts.json

This repo includes a generator that produces a multi-turn ChatML dataset that:

  • asks questions only for blocking unclear spots
  • continues immediately after the user answers
  • avoids invalid samples (ChatML tags are never sliced during presplitting)
python3 tools/build_prompt_dataset.py

Outputs (default):

  • ~/.lokumai/lora_data/train.jsonl
  • ~/.lokumai/lora_data/valid.jsonl

Change dataset size:

LOKUMAI_PROMPT_DATASET_SIZE=20000 python3 tools/build_prompt_dataset.py

Project layout

Core modules:

  • main.py — UI, streaming, persistence, Dev tools
  • rag_engine.py — ingestion + indexing + retrieval
  • file_ingest.py — extraction + chunking (PDF/DOCX/ZIM/OCR)
  • finetune_engine.py — MLX LoRA runner + ChatML-aware presplit
  • lokum_paths.py — centralized path & secrets management

Security & privacy

  • The app is local-first: chats, RAG, and LoRA artifacts are stored under ~/.lokumai/ by default.
  • Repo .gitignore is configured to ignore sensitive/large artifacts (DBs, datasets, adapters, binaries).
  • See internal notes: INTERNAL_SECURITY.md

Troubleshooting

Model path not found

  • Set model_path in prompts.json to a valid MLX model folder.

RAG engine not available

  • Install requirements and ensure:
    • sentence-transformers
    • faiss-cpu

OCR returns empty text

  • brew install tesseract