Add workflow ingestion examples

template/.env.jinja

@@ -1,6 +1,8 @@
 MISTRAL_API_KEY={{ mistral_api_key }}
 MISTRAL_API_URL=https://api.mistral.ai
 COLLECTION_NAME={{ collection_name }}
+# Required for workflow examples (make start-examples / make execute-ingestion)
+DEPLOYMENT_NAME={{ _copier_conf.dst_path.name }}
 WORKSPACE_ROOT=.
 # Optional: change only if ports 18080 / 19072 are already in use
 VESPA_QUERY_PORT=18080

template/Makefile.jinja

@@ -1,4 +1,4 @@
-.PHONY: installdeps setup-vespa start-vespa stop-vespa reset-vespa migrate-vespa verify-vespa ingest search bruno generate-vespa-lock
+.PHONY: installdeps install-workflows setup-vespa start-vespa stop-vespa reset-vespa migrate-vespa verify-vespa ingest search bruno generate-vespa-lock start-examples execute-ingestion
 
 ifneq (,$(wildcard .env))
 include .env
@@ -65,3 +65,16 @@ generate-vespa-lock:
 	uv run mistral-vespa generate \
 		--app-dir src/vespa_app \
 		--path ./vespa.lock
+
+## Install optional workflows dependency (required for examples/workflows/)
+install-workflows:
+	uv sync --extra workflows
+
+## Start a worker that registers the example workflows (requires install-workflows)
+start-examples: install-workflows
+	uv run python -m examples.workflows.worker
+
+## Execute the ingestion workflow via the Mistral Workflows API
+## Usage: make execute-ingestion input='{"file_path": "sample_data/hello.txt", "collection_name": "mydocs"}'
+execute-ingestion: install-workflows
+	uv run python -m examples.workflows.start --workflow document-ingestion $(if $(input),--input '$(input)',--input '{"file_path":"sample_data/hello.txt"}')

template/pyproject.toml.jinja

@@ -8,6 +8,12 @@ dependencies = [
     "python-dotenv>=0.9.9",
 ]
 
+[project.optional-dependencies]
+workflows = [
+    "mistralai-workflows[mistralai]>=3.0.0,<4",
+    "pydantic",
+]
+
 [dependency-groups]
 dev = [
     "ruff>=0.11",
@@ -16,6 +22,9 @@ dev = [
 
 [tool.uv]
 exclude-newer = "7 days"
+# Gemfury (UV_EXTRA_INDEX_URL) may list older versions of mistral-common /
+# mistralai-search-toolkit than PyPI. Allow uv to pick the best match across indexes.
+index-strategy = "unsafe-best-match"
 
 [tool.uv.exclude-newer-package]
 mistralai = false
@@ -28,4 +37,4 @@ requires = ["hatchling"]
 build-backend = "hatchling.build"
 
 [tool.hatch.build.targets.wheel]
-packages = ["src/entrypoints", "src/vespa_app"]
+packages = ["src/entrypoints", "src/vespa_app", "src/examples"]

template/src/examples/__init__.py

@@ -0,0 +1,7 @@
+"""Examples for the search-starter-app.
+
+Workflow-based examples live under ``examples/workflows/`` — see that folder's
+README for setup and usage.
+
+Other examples that do not use the workflows framework live directly here.
+"""

template/src/examples/workflows/README.md

@@ -0,0 +1,133 @@
+# Workflow Examples
+
+This folder contains examples that integrate the search-starter-app with the [Mistral Workflows](https://docs.mistral.ai/capabilities/workflows/) framework.
+
+Workflow-based examples live under `examples/workflows/`. Other examples that do not use the workflows framework (standalone scripts, direct API calls, etc.) live directly under `examples/`.
+
+## Available example
+
+| Example | Workflow name | Makefile target | Description |
+| --- | --- | --- | --- |
+| [search/](search/) | `document-ingestion` | `make execute-ingestion` | Ingestion workflow with small Temporal I/O |
+
+The example ingests local files into Vespa using the same Search Toolkit components as `make ingest`, wrapped in a durable workflow.
+
+## Prerequisites
+
+The workflows framework is an **optional** dependency — the core search project works without it.
+
+> Run these commands from your **generated project root** (the folder created by `copier copy`), not from the `search-starter-app` template repo itself.
+
+```bash
+make install-workflows   # uv sync --extra workflows
+make setup-vespa
+```
+
+Ensure these are set in your `.env` file:
+
+- `MISTRAL_API_KEY`
+- `DEPLOYMENT_NAME` — a stable identifier for this worker (defaults to your project name when generated via `copier copy`)
+
+## Workflow input
+
+```json
+{
+  "file_path": "sample_data/hello.txt",
+  "collection_name": "exampledocs"
+}
+```
+
+| Field | Required | Default | Description |
+| --- | --- | --- | --- |
+| `file_path` | yes | — | Path to a file or directory to ingest |
+| `collection_name` | no | `"exampledocs"` | Vespa collection name |
+
+## How to run
+
+> **Important:** execution commands call the Mistral Workflows API. The workflow must be registered first by a running worker. If you see `Workflow not found`, start the worker below and retry.
+
+**Terminal 1 — start the examples worker (leave this running):**
+
+```bash
+make start-examples
+```
+
+Wait until the worker is listening for execution requests.
+
+**Terminal 2 — trigger the workflow:**
+
+```bash
+make execute-ingestion
+make execute-ingestion input='{"file_path": "sample_data/hello.txt"}'
+make execute-ingestion input='{"file_path": "sample_data", "collection_name": "mydocs"}'
+```
+
+You can also trigger the workflow from the [Mistral Console](https://console.mistral.ai/build/workflows): select `document-ingestion`, click **Start Workflow**, and provide the input JSON.
+
+After ingestion, search directly (no workflow):
+
+```bash
+make search query="hello world"
+```
+
+## Folder layout
+
+```text
+examples/workflows/
+├── README.md              # This file
+├── worker.py              # Registers and runs all workflow examples
+├── start.py               # CLI to trigger a workflow execution
+└── search/                # Document ingestion workflow
+    ├── models.py
+    ├── activities.py
+    ├── workflow.py
+    └── README.md
+```
+
+## Design principles
+
+- **Workflows for ingestion** — document ingestion is long-running and benefits from durability, retries, and observability in the Mistral Console.
+- **Search stays direct** — search queries use `make search` / `entrypoints/search.py` for low latency. Workflows are not needed for queries.
+- **Small activity I/O** — the workflow passes file paths and collection names only. The full Search Toolkit pipeline runs inside a single `ingest_documents` activity; the activity returns a small result (`total_chunks`, `file_count`, …).
+- **Activities own all I/O** — filesystem access, API calls, and Vespa writes live in `activities.py`. The workflow body in `workflow.py` only orchestrates.
+
+## Designing ingestion workflows at scale
+
+Temporal stores every activity input and output in Postgres. **Do not split ingestion into one activity per pipeline stage** if each step passes document bytes, text, chunks, or embeddings — that duplicates large payloads in workflow history and does not scale.
+
+| Step | Cost | Where results should live |
+| --- | --- | --- |
+| Load from remote source | Medium | Object storage (S3) keyed by source URI + version |
+| Extract plain text, split | Cheap | Recompute inside the activity on retry — do not cache in Temporal |
+| OCR, embeddings | Expensive | External cache (Redis) keyed by `hash(file_bytes)`, not the raw content — implement outside this starter example |
+
+**Anti-pattern:** one `@workflows.activity` per pipeline stage with serialized `File` / `Document` dicts passed between them. That pattern only works for tiny demo files and fills Temporal storage quickly.
+
+### Per-stage retries
+
+This starter example runs the full pipeline inside a single activity so Temporal only stores small inputs and outputs (paths in, chunk counts out).
+
+If you need to **split the pipeline into separate activities** — for example, independent retry policies on OCR or embedding — you must still keep activity arguments and return values small. Pass references (file paths, S3 URIs, content hashes) between steps, and store intermediate or expensive-step results in external storage rather than in workflow history.
+
+## Adding a new workflow example
+
+1. Create a subdirectory under `examples/workflows/` (e.g. `my_example/`).
+2. Add `models.py`, `activities.py`, `workflow.py`, and a `README.md`.
+3. Export the workflow class from the package `__init__.py`.
+4. Register it in `EXAMPLE_WORKFLOWS` in `worker.py`.
+5. Add a Makefile target that calls `examples.workflows.start --workflow <name>`.
+
+## Troubleshooting
+
+| Issue | Fix |
+| --- | --- |
+| `Workflow not found` | Start `make start-examples` in a separate terminal, then retry |
+| `DEPLOYMENT_NAME is required` | Add `DEPLOYMENT_NAME=<your-project>` to `.env` |
+| Worker fails on startup | Ensure `imports_passed_through()` wraps activity imports in `workflow.py` when activities use `mistralai.client` |
+| Vespa errors during indexing | Run `make setup-vespa` before triggering ingestion |
+
+Enable verbose logging:
+
+```bash
+LOG_LEVEL=DEBUG make start-examples
+```

template/src/examples/workflows/__init__.py

@@ -0,0 +1,5 @@
+"""Workflow-based examples for the search-starter-app.
+
+Each subdirectory is a self-contained workflow example. Run the worker with:
+    make start-examples
+"""

template/src/examples/workflows/search/README.md

@@ -0,0 +1,75 @@
+# Document Ingestion Workflow Example
+
+This example wraps the search-starter-app ingestion pipeline in a [Mistral Workflow](https://docs.mistral.ai/capabilities/workflows/), adding durability, observability, and retry support.
+
+It follows the recommended pattern for ingestion at scale: **small activity I/O**, with the full Search Toolkit pipeline running inside a single activity.
+
+## What It Demonstrates
+
+| Primitive | Where |
+| --- | --- |
+| `@workflows.workflow.define` | `workflow.py` — workflow registered in the Mistral Console |
+| `@workflows.workflow.entrypoint` | `workflow.py` — orchestration only; passes paths, not document content |
+| `@workflows.activity` | `activities.py` — all file I/O, embedding calls, and Vespa writes |
+| Pydantic input/output models | `models.py` — typed boundaries; workflow output is a small summary |
+| `workflows.run_worker()` | `worker.py` — registers the workflow with Temporal |
+
+## Architecture
+
+```
+IngestionWorkflow.run(IngestionInput)
+├── collect_document_paths activity   → list[str] paths only
+└── ingest_documents activity         → IngestionResult (counts + status)
+    └── Search Toolkit Pipeline:
+        load → extract → split → embed → index
+```
+
+The workflow never sees file bytes, extracted text, chunks, or embeddings — only paths and the final chunk count.
+
+## Prerequisites
+
+1. Install the workflows extra:
+   ```bash
+   make install-workflows
+   ```
+
+2. Vespa must be running:
+   ```bash
+   make setup-vespa
+   ```
+
+3. Set `MISTRAL_API_KEY` and `DEPLOYMENT_NAME` in your `.env` file.
+
+## Running the Example
+
+> The worker must be running before you trigger the workflow, otherwise the API returns `Workflow not found`.
+
+**Terminal 1 — start the worker (leave this running):**
+```bash
+make start-examples
+```
+
+**Terminal 2 — trigger ingestion:**
+```bash
+make execute-ingestion input='{"file_path": "sample_data/hello.txt"}'
+make execute-ingestion input='{"file_path": "sample_data"}'
+make execute-ingestion input='{"file_path": "sample_data/hello.txt", "collection_name": "mydocs"}'
+```
+
+You can also trigger the workflow from the [Mistral Console](https://console.mistral.ai/build/workflows) by selecting `document-ingestion`.
+
+## What not to do
+
+Do **not** split ingestion into one activity per pipeline stage while passing serialized documents between them. Temporal persists every activity input/output in Postgres; that pattern duplicates large payloads and does not scale. See the parent [workflows README](../README.md#designing-ingestion-workflows-at-scale).
+
+If you want to split the pipeline to enable retries for some stages, follow the [Handling Large Data](https://docs.mistral.ai/capabilities/workflows/guides/handling-large-data/) guide instead of passing document content between activities.
+
+## Key Difference from Direct Ingestion
+
+`make ingest` runs the same Search Toolkit pipeline synchronously. The workflow adds:
+
+- **Durability** — the worker can restart mid-ingestion and resume from the last completed activity
+- **Observability** — executions appear in the Mistral Console
+- **Retries** — the activity retries on transient errors without bloating Temporal with per-step document payloads
+
+Search queries remain direct (`make search`) — they do not need workflow orchestration.

template/src/examples/workflows/search/__init__.py

@@ -0,0 +1,9 @@
+"""Workflows × Search integration example.
+
+Demonstrates wrapping the search-starter-app ingestion pipeline in a
+Mistral Workflow for durability, observability, and HITL support.
+"""
+
+from .workflow import IngestionWorkflow
+
+__all__ = ["IngestionWorkflow"]