feat: ElevenLabs Scribe backend with speaker diarization

README.md

@@ -49,10 +49,37 @@ trx transcribe video.mp4 --language es
 # Keep fillers and false starts instead of a cleaned-up transcript
 trx transcribe video.mp4 --words --language es --preset verbatim
 
+# Separate speakers (ElevenLabs Scribe)
+trx transcribe interview.m4a -b elevenlabs --speakers 2
+
 # Schema introspection for agents
 trx schema transcribe
 ```
 
+### Speaker diarization
+
+`--backend elevenlabs` transcribes through ElevenLabs Scribe, the one backend that reports who
+is speaking. `--diarize` prefixes every cue with its speaker; `--speakers <n>` adds the count
+when you know it and implies `--diarize`.
+
+```srt
+1
+00:00:01,900 --> 00:00:03,400
+[speaker_0] Hola, que tal? Escuchas?
+
+2
+00:00:04,520 --> 00:00:05,900
+[speaker_1] Si, te escucho bien.
+```
+
+Scribe timestamps every word, so cues are grouped to stay readable: a pause of 0.6s or more
+starts a new cue, a cue is capped at 84 characters, and a change of speaker always starts a new
+one so no single cue attributes two people to one line. The `.txt` reads as a conversation, one
+paragraph per turn. Both flags are rejected on other backends rather than ignored.
+
+Needs `ELEVENLABS_API_KEY`. On macOS trx also reads the `elevenlabs` entry from your login
+Keychain, so a key stored there does not have to be exported into every shell.
+
 ### Verbatim transcripts
 
 A transcriber cleans by default: it writes what it believes was meant, so hesitations,
@@ -112,7 +139,7 @@ Input (URL or file)
 [ffmpeg] Clean audio (noise reduction, normalization; duration preserved)
   |
   v
-[whisper-cli] Transcribe (local Whisper model)
+[whisper-cli | OpenAI | Vercel AI Gateway | ElevenLabs Scribe] Transcribe
   |
   v
 Output: .wav + .srt + .txt + JSON
@@ -124,13 +151,17 @@ Stored at `~/.trx/config.json` after `trx init`:
 
 ```json
 {
+  "backend": "local",
   "modelPath": "~/.trx/models/ggml-small.bin",
   "modelSize": "small",
   "language": "auto",
-  "threads": 8
+  "threads": 8,
+  "elevenlabs": { "model": "scribe_v2", "diarize": false }
 }
 ```
 
+Backends: `local` (whisper.cpp) | `openai` (`OPENAI_API_KEY`) | `vercel` (`AI_GATEWAY_API_KEY`) | `elevenlabs` (`ELEVENLABS_API_KEY`, speaker diarization)
+
 Models: `tiny` (75MB) | `base` (142MB) | `small` (466MB) | `medium` (1.5GB) | `large` (3GB)
 
 ## License

packages/cli/AGENTS.md

@@ -1,6 +1,6 @@
 # @crafter/trx - Agent Guide
 
-`@crafter/trx` is an agent-first CLI that transcribes audio/video from URLs (YouTube, Twitter, Instagram, any yt-dlp source) or local files using Whisper, locally via whisper-cli or remotely via the OpenAI API. Reach for it when a task needs speech-to-text: extracting the transcript of a video/podcast, generating subtitles (.srt), or turning a recording into text for summarization. Output auto-switches to JSON when piped, supports `--fields` to limit payload size, `--dry-run` validation, and `trx schema` introspection.
+`@crafter/trx` is an agent-first CLI that transcribes audio/video from URLs (YouTube, Twitter, Instagram, any yt-dlp source) or local files using Whisper, locally via whisper-cli or remotely via the OpenAI API, the Vercel AI Gateway, or ElevenLabs Scribe (the one backend that separates speakers). Reach for it when a task needs speech-to-text: extracting the transcript of a video/podcast, generating subtitles (.srt), or turning a recording into text for summarization. Output auto-switches to JSON when piped, supports `--fields` to limit payload size, `--dry-run` validation, and `trx schema` introspection.
 
 ## Install
 
@@ -32,7 +32,7 @@ Global option (all commands): `-o, --output <format>` with `json`, `table`, or `
 
 | Flag | Description |
 |------|-------------|
-| `-b, --backend <backend>` | `local` (whisper-cli) or `openai` (needs `OPENAI_API_KEY`) |
+| `-b, --backend <backend>` | `local` (whisper-cli), `openai` (needs `OPENAI_API_KEY`), `vercel` (needs `AI_GATEWAY_API_KEY`), or `elevenlabs` (needs `ELEVENLABS_API_KEY`) |
 | `-m, --model <size>` | Whisper model size, default `small` (`tiny`/`base`/`small`/`medium`/`large`) |
 | `-l, --language <code>` | Default language, `auto` = detect |
 
@@ -41,8 +41,10 @@ Global option (all commands): `-o, --output <format>` with `json`, `table`, or `
 | Flag | Description |
 |------|-------------|
 | `-l, --language <lang>` | Force language (ISO 639-1), default auto-detect |
-| `-m, --model <size>` | Override model. Local: `tiny`/`base`/`small`/`medium`/`large`/`large-v3-turbo`. OpenAI: `gpt-4o-transcribe`/`gpt-4o-mini-transcribe`/`whisper-1` |
-| `-b, --backend <backend>` | `local`, `openai`, or `vercel` |
+| `-m, --model <size>` | Override model. Local: `tiny`/`base`/`small`/`medium`/`large`/`large-v3-turbo`. OpenAI: `gpt-4o-transcribe`/`gpt-4o-mini-transcribe`/`whisper-1`. ElevenLabs: `scribe_v2`/`scribe_v1` |
+| `-b, --backend <backend>` | `local`, `openai`, `vercel`, or `elevenlabs` |
+| `--diarize` | Label each cue with its speaker. `elevenlabs` only; an error on other backends |
+| `--speakers <n>` | Expected speaker count, 1-32. Implies `--diarize`. `elevenlabs` only |
 | `--fields <fields>` | Limit output fields: `text`, `srt`, `metadata`, `files` |
 | `--dry-run` | Validate input and show plan without transcribing |
 | `--json <payload>` | Raw JSON input for agents |
@@ -72,6 +74,10 @@ Global option (all commands): `-o, --output <format>` with `json`, `table`, or `
    ```bash
    OPENAI_API_KEY=sk-... trx transcribe interview.m4a --backend openai --model gpt-4o-mini-transcribe
    ```
+5. Transcribe a two-person interview with speakers separated:
+   ```bash
+   trx transcribe interview.m4a --backend elevenlabs --speakers 2 --fields text --output json
+   ```
 
 ## Decision guide
 
@@ -83,12 +89,14 @@ Global option (all commands): `-o, --output <format>` with `json`, `table`, or `
 | Discover flags at runtime without docs | `trx schema transcribe` |
 | Fast cloud transcription, no local model download | `--backend openai` with `OPENAI_API_KEY` set |
 | Force Spanish (or any language) | `trx transcribe <input> --language es` |
+| Know who said what in a meeting or interview | `--backend elevenlabs --diarize` (add `--speakers <n>` when the count is known) |
 
 ## Common mistakes
 
 - Wrong: `npm install trx` (different, unrelated package). Correct: `npm install -g @crafter/trx` or `bunx @crafter/trx`.
 - Wrong: running `trx transcribe` on a fresh machine. Correct: run `trx init` first; it installs whisper-cli/yt-dlp/ffmpeg and downloads the model. `trx doctor` tells you what is missing.
 - Wrong: assuming Node is enough. The `trx` binary has a `#!/usr/bin/env bun` shebang and requires Bun (`engines.bun >= 1.0.0`) even when installed through npm/npx.
 - Wrong: `--backend openai` without credentials. Correct: export `OPENAI_API_KEY` first.
+- Wrong: `--diarize` on `local`, `openai` or `vercel`. No other backend returns speaker labels, so trx rejects the flag instead of returning an undiarized transcript that looks like it worked. Correct: `--backend elevenlabs --diarize`.
 - Wrong: adding `--output json` manually in scripts. Not harmful, but unnecessary: output is already JSON whenever stdout is not a TTY.
 - Wrong: expecting `trx init` to work non-interactively on Linux via Homebrew. Dependency install uses `brew`; without it, install `whisper-cli`, `yt-dlp`, and `ffmpeg` manually, then re-run `trx init` for the model download.

packages/cli/schemas/init.json

@@ -1,21 +1,22 @@
 {
   "command": "init",
-  "description": "Install dependencies and configure transcription backend (local whisper-cli, OpenAI API, or Vercel AI Gateway)",
+  "description": "Install dependencies and configure transcription backend (local whisper-cli, OpenAI API, Vercel AI Gateway, or ElevenLabs Scribe)",
   "flags": {
     "--backend": {
       "type": "string",
       "enum": [
         "local",
         "openai",
-        "vercel"
+        "vercel",
+        "elevenlabs"
       ],
       "default": "local",
-      "description": "Transcription backend. 'openai' and 'vercel' skip whisper-cli install; 'openai' requires OPENAI_API_KEY, 'vercel' requires AI_GATEWAY_API_KEY"
+      "description": "Transcription backend. 'openai', 'vercel' and 'elevenlabs' skip whisper-cli install; 'openai' requires OPENAI_API_KEY, 'vercel' requires AI_GATEWAY_API_KEY, 'elevenlabs' requires ELEVENLABS_API_KEY"
     },
     "--model": {
       "type": "string",
       "default": "small",
-      "description": "Model to use. Local: tiny, base, small, medium, large, large-v3-turbo. OpenAI: gpt-4o-transcribe, gpt-4o-mini-transcribe, whisper-1. Vercel: creator/model-name, e.g. openai/whisper-1"
+      "description": "Model to use. Local: tiny, base, small, medium, large, large-v3-turbo. OpenAI: gpt-4o-transcribe, gpt-4o-mini-transcribe, whisper-1. Vercel: creator/model-name, e.g. openai/whisper-1. ElevenLabs: scribe_v2, scribe_v1"
     },
     "--language": {
       "type": "string",
@@ -36,7 +37,7 @@
   "dependencies": {
     "whisper-cli": {
       "install": "brew install whisper-cpp",
-      "purpose": "Local speech-to-text transcription engine (not needed for openai or vercel backends)"
+      "purpose": "Local speech-to-text transcription engine (not needed for openai, vercel or elevenlabs backends)"
     },
     "yt-dlp": {
       "install": "brew install yt-dlp",
@@ -49,7 +50,7 @@
   },
   "output": {
     "success": "boolean",
-    "backend": "string (local | openai | vercel)",
+    "backend": "string (local | openai | vercel | elevenlabs)",
     "model": "string",
     "language": "string",
     "modelPath": "string (local only)",
@@ -60,6 +61,7 @@
     "trx init --backend openai",
     "trx init --backend openai --model gpt-4o-mini-transcribe",
     "trx init --backend vercel --model openai/whisper-1",
+    "trx init --backend elevenlabs --model scribe_v2",
     "trx init --model large-v3-turbo --language es",
     "trx init --model large --output json"
   ]

packages/cli/schemas/models.json

@@ -1,13 +1,14 @@
 {
 	"command": "models",
-	"description": "List available transcription models for local, OpenAI, and Vercel AI Gateway backends",
+	"description": "List available transcription models for local, OpenAI, Vercel AI Gateway, and ElevenLabs backends",
 	"flags": {
 		"--backend": {
 			"type": "string",
 			"enum": [
 				"local",
 				"openai",
-				"vercel"
+				"vercel",
+				"elevenlabs"
 			],
 			"description": "Filter models to one backend"
 		},
@@ -25,11 +26,13 @@
 	"output": {
 		"local": "string[]",
 		"openai": "string[]",
+		"elevenlabs": "string[]",
 		"vercel": "string[] or { error: string }"
 	},
 	"examples": [
 		"trx models",
 		"trx models --backend local --output json",
-		"trx models --backend vercel --output json"
+		"trx models --backend vercel --output json",
+		"trx models --backend elevenlabs --output json"
 	]
 }

packages/cli/schemas/transcribe.json

@@ -1,6 +1,6 @@
 {
   "command": "transcribe",
-  "description": "Transcribe audio/video from URL or local file using Whisper (local), OpenAI API, or Vercel AI Gateway",
+  "description": "Transcribe audio/video from URL or local file using Whisper (local), OpenAI API, Vercel AI Gateway, or ElevenLabs Scribe",
   "arguments": {
     "input": {
       "type": "string",
@@ -14,9 +14,19 @@
       "enum": [
         "local",
         "openai",
-        "vercel"
+        "vercel",
+        "elevenlabs"
       ],
-      "description": "Transcription backend. 'local' uses whisper-cli, 'openai' uses OpenAI API (requires OPENAI_API_KEY), 'vercel' uses Vercel AI Gateway (requires AI_GATEWAY_API_KEY)"
+      "description": "Transcription backend. 'local' uses whisper-cli, 'openai' uses OpenAI API (requires OPENAI_API_KEY), 'vercel' uses Vercel AI Gateway (requires AI_GATEWAY_API_KEY), 'elevenlabs' uses ElevenLabs Scribe (requires ELEVENLABS_API_KEY, the only backend with speaker diarization)"
+    },
+    "--diarize": {
+      "type": "boolean",
+      "default": false,
+      "description": "Separate speakers and prefix each SRT cue with its speaker label, e.g. [speaker_0]. A change of speaker always starts a new cue. Requires --backend elevenlabs; passing it to any other backend is an error rather than a silently undiarized transcript."
+    },
+    "--speakers": {
+      "type": "integer",
+      "description": "How many speakers to expect, 1-32 (upper bound set by the API). Implies --diarize. Requires --backend elevenlabs."
     },
     "--language": {
       "type": "string",
@@ -25,7 +35,7 @@
     },
     "--model": {
       "type": "string",
-      "description": "Override model. Local: tiny, base, small, medium, large, large-v3-turbo. OpenAI: gpt-4o-transcribe, gpt-4o-mini-transcribe, whisper-1. Vercel: creator/model-name, e.g. openai/whisper-1"
+      "description": "Override model. Local: tiny, base, small, medium, large, large-v3-turbo. OpenAI: gpt-4o-transcribe, gpt-4o-mini-transcribe, whisper-1. Vercel: creator/model-name, e.g. openai/whisper-1. ElevenLabs: scribe_v2, scribe_v1"
     },
     "--words": {
       "type": "boolean",
@@ -64,7 +74,7 @@
     },
     "--json": {
       "type": "string",
-      "description": "Raw JSON payload: {\"input\": \"...\", \"language\": \"...\", \"model\": \"...\", \"backend\": \"...\", \"cookiesFromBrowser\": \"chrome\"}"
+      "description": "Raw JSON payload: {\"input\": \"...\", \"language\": \"...\", \"model\": \"...\", \"backend\": \"...\", \"diarize\": true, \"speakers\": 2, \"cookiesFromBrowser\": \"chrome\"}"
     },
     "--output-dir": {
       "type": "string",
@@ -84,7 +94,7 @@
     "--no-chunk": {
       "type": "boolean",
       "default": false,
-      "description": "Disable automatic chunking for oversized OpenAI and Vercel uploads"
+      "description": "Disable automatic chunking for oversized OpenAI and Vercel uploads. The elevenlabs backend has a 5 GB limit and never chunks."
     },
     "--cookies-from-browser": {
       "type": "string",
@@ -94,7 +104,7 @@
   "output": {
     "success": "boolean",
     "input": "string",
-    "backend": "string (local | openai | vercel)",
+    "backend": "string (local | openai | vercel | elevenlabs)",
     "files": {
       "wav": "string (path)",
       "srt": "string (path)",
@@ -107,7 +117,7 @@
       "transcribedDurationMs": "integer|null, how long the audio that reached the model ran. A large gap from inputDurationMs means the timeline was rewritten",
       "lastCueEndMs": "integer|null, where the last cue ends. The gap to transcribedDurationMs is audio that produced no words"
     },
-    "text": "string (full transcript)"
+    "text": "string (full transcript; with --diarize, turns are prefixed [speaker_N] and separated by blank lines)"
   },
   "examples": [
     "trx transcribe recording.mp4 --output json",
@@ -117,6 +127,8 @@
     "trx transcribe https://youtube.com/watch?v=abc --language es --output json",
     "trx transcribe https://www.instagram.com/reel/... --cookies-from-browser chrome --output json",
     "trx transcribe video.mp4 --fields text --output json",
-    "trx transcribe video.mp4 --dry-run --output json"
+    "trx transcribe video.mp4 --dry-run --output json",
+    "trx transcribe interview.m4a -b elevenlabs --diarize --output json",
+    "trx transcribe interview.m4a -b elevenlabs --speakers 2 --language spa --output json"
   ]
 }

packages/cli/skills/trx/SKILL.md

@@ -91,6 +91,10 @@ trx transcribe <input> --backend openai
 # Vercel AI Gateway (requires AI_GATEWAY_API_KEY)
 export AI_GATEWAY_API_KEY=...
 trx transcribe <input> --backend vercel --model openai/whisper-1
+
+# ElevenLabs Scribe (requires ELEVENLABS_API_KEY, separates speakers)
+export ELEVENLABS_API_KEY=...
+trx transcribe <input> --backend elevenlabs --diarize
 ```
 
 OpenAI models:
@@ -100,9 +104,22 @@ OpenAI models:
 
 Vercel model IDs use `creator/model-name` format. The default is `openai/whisper-1`. One `AI_GATEWAY_API_KEY` covers every provider on the gateway. Use a transcription model returned by `trx models --backend vercel`; this is Vercel AI Gateway, not Cloudflare AI Gateway.
 
+ElevenLabs models: `scribe_v2` (default) and `scribe_v1`. This is the only backend that reports who is speaking. On macOS the key may live in the login Keychain under `elevenlabs` instead of the environment; `trx doctor` resolves it the same way a run does, so trust its verdict over `echo $ELEVENLABS_API_KEY`.
+
+#### Speaker diarization
+
+```bash
+trx transcribe interview.m4a --backend elevenlabs --diarize --output json
+trx transcribe interview.m4a --backend elevenlabs --speakers 2 --language spa --output json
+```
+
+`--diarize` prefixes every SRT cue with its speaker (`[speaker_0]`) and turns the `.txt` into a conversation, one paragraph per turn. `--speakers <n>` (1-32) passes the count when it is known and implies `--diarize`. A change of speaker always starts a new cue, so no cue attributes two people to one line.
+
+Both flags are an error on `local`, `openai` and `vercel` rather than being ignored: no other backend returns speaker labels, and silently dropping them would hand back an undiarized transcript that looks like the request succeeded. `--language` accepts ISO 639-1 or ISO 639-3 here (`es` and `spa` both work); every other backend takes 639-1 only.
+
 Local models: `tiny`, `tiny.en`, `base`, `base.en`, `small`, `small.en`, `medium`, `medium.en`, `large`, `large-v3-turbo`.
 
-Set the backend persistently with `trx init --backend vercel` (or `local`/`openai`) or in config.
+Set the backend persistently with `trx init --backend vercel` (or `local`/`openai`/`elevenlabs`) or in config.
 
 #### Model discovery
 
@@ -111,13 +128,14 @@ trx models
 trx models --backend local --output json
 trx models --backend openai --output json
 trx models --backend vercel --output json
+trx models --backend elevenlabs --output json
 ```
 
-Local and OpenAI model lists are static. Vercel models are fetched live from the gateway and filtered to transcription models, so do not hard-code that list. `AI_GATEWAY_API_KEY` is required when requesting only the Vercel backend. Without it, the all-backends view still returns local and OpenAI models plus a Vercel error.
+Local, OpenAI and ElevenLabs model lists are static. Vercel models are fetched live from the gateway and filtered to transcription models, so do not hard-code that list. `AI_GATEWAY_API_KEY` is required when requesting only the Vercel backend. Without it, the all-backends view still returns local and OpenAI models plus a Vercel error.
 
 #### Automatic cloud-file chunking
 
-OpenAI uploads over 25 MB and Vercel uploads over 100 MB are chunked automatically with ffmpeg. trx transcribes chunks sequentially, joins their text in order, offsets and renumbers SRT timestamps, and removes intermediate chunk files. Use `--no-chunk` to disable this behavior and fail on an oversized cloud upload.
+OpenAI uploads over 25 MB and Vercel uploads over 100 MB are chunked automatically with ffmpeg. The ElevenLabs limit is 5 GB, so that backend never chunks. trx transcribes chunks sequentially, joins their text in order, offsets and renumbers SRT timestamps, and removes intermediate chunk files. Use `--no-chunk` to disable this behavior and fail on an oversized cloud upload.
 
 ### 3. Post-process (fix whisper mistakes)