You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5eac280
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: packages/ai/README.md
+75-87Lines changed: 75 additions & 87 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,7 +1,7 @@
1
1
# @opencode/ai
2
2
3
-
Schema-first APIs for text, images, decisions, and embeddings, built with Effect. Video, speech, transcription, extraction, and
4
-
realtime voice are [experimental](#experimental).
3
+
Schema-first APIs for text, images, video, decisions, and embeddings, built with Effect. Speech, transcription,
4
+
extraction, and realtime voice are [experimental](#experimental).
5
5
6
6
`@opencode/ai` is the Promise API: one managed runtime behind the LLM and media methods, plus asset helpers. `ai.file`
7
7
and `ai.write` load `node:fs/promises` on first use, so no Effect `FileSystem` is needed:
@@ -211,7 +211,7 @@ Raw usage is preserved in `usage.providerMetadata.google`. `inputTokens` follows
211
211
`contextTokens` uses its full `raw_prompt_token` count when supplied. These can differ substantially with server-side
212
212
continuation. Explicit caches, hosted tools, and generated media are not supported by this initial protocol.
213
213
214
-
The same configured facade names image models (and the experimental video, speech, and transcription models).
214
+
The same configured facade names image and video models (and the experimental speech and transcription models).
215
215
`ai.image.generate` resolves the provider's image route from the model and returns `Media.Asset`s with lazily decoded bytes:
216
216
217
217
```ts
@@ -1031,6 +1031,72 @@ const program = Effect.gen(function* () {
1031
1031
1032
1032
The hosted result is represented as a provider-executed tool call and a tool result whose content carries the generated image as a file. Gemini image-capable models instead emit a first-class `media``LLMEvent` for inline image output (`response.message` then carries a `media` part). Retaining `response.message` preserves the generated image for continuation on both routes.
1033
1033
1034
+
## Video generation
1035
+
1036
+
Video generation is a job: `start` submits it and returns at once, and the caller decides when to read its `status`,
1037
+
fetch its `result`, or `cancel` it. Nothing polls for you. Every call after `start` takes the model and the job's
1038
+
`token`, plain JSON you can store and read back from another process, so recovering after a restart is the same call
1039
+
as a normal status read. Models come from `.video(...)` selectors on the `XAI` facade.
1040
+
1041
+
```ts
1042
+
import { Effect } from"effect"
1043
+
import { ai, Media } from"@opencode/ai/effect"
1044
+
import { XAI } from"@opencode/ai/providers"
1045
+
1046
+
const model =XAI.configure({ apiKey: process.env.XAI_API_KEY }).video("grok-imagine-video-1.5")
1047
+
1048
+
const program =Effect.gen(function* () {
1049
+
const job =yield*ai.video.start({
1050
+
model,
1051
+
prompt: "Panning wide shot of a calico kitten sleeping in the sunshine",
1052
+
start: "https://example.com/first-frame.png", // exact first frame; `end` pins the last one
1053
+
references: [], // images, videos or audio that guide subject or style
1054
+
duration: "8 seconds",
1055
+
aspect: "16:9", // or "auto" to follow the start image
1056
+
resolution: "720p", // 480p | 720p | 1080p | 4k, or any provider value
1057
+
sound: false,
1058
+
})
1059
+
job.token// `{ requestID }` for xAI: store it with the provider and model ID
1060
+
1061
+
const status =yield*ai.video.status(model, job.token)
-**`ai.image.request` / `generate` / `stream` / `start` / `resume`** — images over inline, streaming (partial previews), and queued routes through a provider-neutral request and response model.
1046
1112
-**`ImageClient`** — Effect service and layer for image execution, parallel to `LLMClient`.
1113
+
-**`ai.video.request` / `start` / `status` / `result` / `cancel`** — video generation as a job addressed by a stored token; see [Video generation](#video-generation). `VideoClient` is its Effect service and layer.
1047
1114
-**`Media`** — the shared asset type (`Media.Asset`, `Media.Source`) and constructors used by messages, tool results, and media requests.
1048
1115
-**`Generation`** — provider-neutral handle for an in-flight media generation (`await`, `refresh`, `cancel`, `events`) used by queued media routes.
1049
1116
-**`ai.decision.request` / `ai.decision.generate`** — typed predicate, choice, and score questions over text, JSON, and media input; `DecisionClient` is its Effect service and layer.
1050
1117
-**`ai.embedding.request` / `ai.embedding.generate`** — one vector per text or media value, with optional `dimensions` and `inputType`; `EmbeddingClient` is its Effect service and layer.
1051
1118
-**`AIClient.layer` / `AIClient.layerWith(executor)`** — every modality client plus the request executor in one layer.
1052
-
-**`@opencode/ai`** (Promise root) — a default `ai` with the text calls (`request`, `generate`, `stream`, `loop`, `compact`, `canCompact`) at the top level plus `image`, `decision` and `embedding`(plus `experimental.video`, `experimental.speech`, and `experimental.transcription`) as Promise / `AsyncIterable` APIs, `run(effect)` for any package Effect, plus `file`, `write`, `bytes`, `base64`, and `materialize` for assets. `AI.make({ layer? })` builds a client over a custom executor. The same `ai` from `@opencode/ai/effect` returns `Effect` and `Stream`.
1053
-
-**`@opencode/ai/experimental`** — APIs that may change without compatibility guarantees: `Video`, `Speech`, `Transcription`, `Extraction`, and `Realtime`. See [Experimental](#experimental).
1119
+
-**`@opencode/ai`** (Promise root) — a default `ai` with the text calls (`request`, `generate`, `stream`, `loop`, `compact`, `canCompact`) at the top level plus `image`, `decision`, `embedding`and `video` (plus `experimental.speech` and `experimental.transcription`) as Promise / `AsyncIterable` APIs, `run(effect)` for any package Effect, plus `file`, `write`, `bytes`, `base64`, and `materialize` for assets. `AI.make({ layer? })` builds a client over a custom executor. The same `ai` from `@opencode/ai/effect` returns `Effect` and `Stream`.
1120
+
-**`@opencode/ai/experimental`** — APIs that may change without compatibility guarantees: `Speech`, `Transcription`, `Extraction`, and `Realtime`. See [Experimental](#experimental).
1054
1121
1055
1122
## Experimental
1056
1123
1057
1124
These APIs may change without compatibility guarantees. The Effect API is exported from `@opencode/ai/experimental`,
1058
1125
and the Promise client exposes it under `ai.experimental`.
1059
1126
1060
-
-**`Video`**, **`Speech`**, and **`Transcription`** — media generation and speech-to-text, with `VideoClient`,
1061
-
`SpeechClient`, and `TranscriptionClient` as their Effect services. Promise: `ai.experimental.video`, `.speech`, and
1062
-
`.transcription`.
1127
+
-**`Speech`** and **`Transcription`** — speech generation and speech-to-text, with `SpeechClient` and
1128
+
`TranscriptionClient` as their Effect services. Promise: `ai.experimental.speech` and `.transcription`.
1063
1129
-**`Extraction`** — typed values read out of text.
1064
1130
-**`Realtime`** — `Realtime.connect` starts an OpenAI, xAI, or Gemini voice session and returns a WebRTC answer SDP or
1065
1131
a short-lived WebSocket URL and token.
1066
1132
1067
-
The provider selectors for these modalities (`.video(...)`, `.speech(...)`, `.transcription(...)`,
1133
+
The provider selectors for these modalities (`.speech(...)`, `.transcription(...)`,
1068
1134
`.experimental.extraction(...)`) and their protocol entrypoints (such as `@opencode/ai/protocols/openai-speech`) are
1069
1135
part of the same experimental surface.
1070
1136
1071
-
### Video generation
1072
-
1073
-
Video mirrors `Image` with one difference: every provider is asynchronous, so the route is a submit-then-poll
1074
-
`Generation`. Models come from `.video(...)` selectors on the `XAI` facade.
0 commit comments