Skip to content

Commit 5eac280

Browse files
authored
feat(ai): graduate video generation with start, status, result and cancel (#63657)
1 parent 265be7a commit 5eac280

18 files changed

Lines changed: 759 additions & 807 deletions

‎packages/ai/README.md‎

Lines changed: 75 additions & 87 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# @opencode/ai
22

3-
Schema-first APIs for text, images, decisions, and embeddings, built with Effect. Video, speech, transcription, extraction, and
4-
realtime voice are [experimental](#experimental).
3+
Schema-first APIs for text, images, video, decisions, and embeddings, built with Effect. Speech, transcription,
4+
extraction, and realtime voice are [experimental](#experimental).
55

66
`@opencode/ai` is the Promise API: one managed runtime behind the LLM and media methods, plus asset helpers. `ai.file`
77
and `ai.write` load `node:fs/promises` on first use, so no Effect `FileSystem` is needed:
@@ -211,7 +211,7 @@ Raw usage is preserved in `usage.providerMetadata.google`. `inputTokens` follows
211211
`contextTokens` uses its full `raw_prompt_token` count when supplied. These can differ substantially with server-side
212212
continuation. Explicit caches, hosted tools, and generated media are not supported by this initial protocol.
213213

214-
The same configured facade names image models (and the experimental video, speech, and transcription models).
214+
The same configured facade names image and video models (and the experimental speech and transcription models).
215215
`ai.image.generate` resolves the provider's image route from the model and returns `Media.Asset`s with lazily decoded bytes:
216216

217217
```ts
@@ -1031,6 +1031,72 @@ const program = Effect.gen(function* () {
10311031

10321032
The hosted result is represented as a provider-executed tool call and a tool result whose content carries the generated image as a file. Gemini image-capable models instead emit a first-class `media` `LLMEvent` for inline image output (`response.message` then carries a `media` part). Retaining `response.message` preserves the generated image for continuation on both routes.
10331033

1034+
## Video generation
1035+
1036+
Video generation is a job: `start` submits it and returns at once, and the caller decides when to read its `status`,
1037+
fetch its `result`, or `cancel` it. Nothing polls for you. Every call after `start` takes the model and the job's
1038+
`token`, plain JSON you can store and read back from another process, so recovering after a restart is the same call
1039+
as a normal status read. Models come from `.video(...)` selectors on the `XAI` facade.
1040+
1041+
```ts
1042+
import { Effect } from "effect"
1043+
import { ai, Media } from "@opencode/ai/effect"
1044+
import { XAI } from "@opencode/ai/providers"
1045+
1046+
const model = XAI.configure({ apiKey: process.env.XAI_API_KEY }).video("grok-imagine-video-1.5")
1047+
1048+
const program = Effect.gen(function* () {
1049+
const job = yield* ai.video.start({
1050+
model,
1051+
prompt: "Panning wide shot of a calico kitten sleeping in the sunshine",
1052+
start: "https://example.com/first-frame.png", // exact first frame; `end` pins the last one
1053+
references: [], // images, videos or audio that guide subject or style
1054+
duration: "8 seconds",
1055+
aspect: "16:9", // or "auto" to follow the start image
1056+
resolution: "720p", // 480p | 720p | 1080p | 4k, or any provider value
1057+
sound: false,
1058+
})
1059+
job.token // `{ requestID }` for xAI: store it with the provider and model ID
1060+
1061+
const status = yield* ai.video.status(model, job.token)
1062+
status.status // "queued" | "running" | "completed" | "failed" | "cancelled" | "expired"
1063+
status.progress // 0..1, only when the provider reports it
1064+
1065+
const response = yield* ai.video.result(model, job.token)
1066+
yield* Media.write(response.video, "./kitten.mp4") // streams the download to disk
1067+
})
1068+
```
1069+
1070+
The shared fields are `prompt`, `start`, `end`, `references`, `duration`, `aspect`, `resolution` and `sound`;
1071+
everything else is provider-specific and goes under `providerOptions`, typed from the selected model. A request only
1072+
fails before sending when its protocol has no way to express it, such as a reference video on xAI. Values the provider
1073+
may reject, like an unsupported duration or too many references, go to the provider, and its error comes back typed.
1074+
1075+
`status` is one read; `result` fails with an `InvalidRequest` reason when the job hasn't finished, and with the
1076+
provider's reason (and terminal document on `reason.body`) when it failed, was cancelled or expired. Moderation
1077+
outcomes (xAI `respect_moderation`) fail with a `ContentPolicy` reason when no video comes back. Status and result reads
1078+
retry transient failures (rate limits, provider 5xx, network errors) with backoff that honors `retry-after`; `start`
1079+
and `cancel` never retry. `cancel` fails with an `UnsupportedOperation` reason on providers without a cancel endpoint,
1080+
such as xAI, where the job keeps running and billing. Finished videos are usually URLs that expire (`expiresAt` when the
1081+
provider documents it): download them with `Media.write`, `asset.stream()` or `materialize()` before then.
1082+
1083+
Provider notes:
1084+
1085+
- **xAI** maps `start` to `image`, `end` to `last_frame`, image references to `reference_images`, audio references to
1086+
`reference_audios`, and `sound` to `generate_audio`, and leaves `aspect: "auto"` out when a start frame sets the
1087+
shape. It has no reference video input. `providerOptions.mode: "edit" | "extend"` sends a native `video: { url }`
1088+
source to `/videos/edits` or `/videos/extensions`.
1089+
1090+
The promise client has the same calls, each accepting `{ signal }`:
1091+
1092+
```ts
1093+
import { ai } from "@opencode/ai"
1094+
1095+
const job = await ai.video.start({ model, prompt })
1096+
const status = await ai.video.status(model, job.token)
1097+
if (status.status === "completed") await ai.write((await ai.video.result(model, job.token)).video, "./kite.mp4")
1098+
```
1099+
10341100
## Public API
10351101

10361102
Everything below is exported from `@opencode/ai/effect` unless noted.
@@ -1044,108 +1110,30 @@ Everything below is exported from `@opencode/ai/effect` unless noted.
10441110
- **`LLMEvent.is.*`** — typed guards (`is.textDelta`, `is.toolCall`, `is.finish`, …) for filtering streams.
10451111
- **`ai.image.request` / `generate` / `stream` / `start` / `resume`** — images over inline, streaming (partial previews), and queued routes through a provider-neutral request and response model.
10461112
- **`ImageClient`** — Effect service and layer for image execution, parallel to `LLMClient`.
1113+
- **`ai.video.request` / `start` / `status` / `result` / `cancel`** — video generation as a job addressed by a stored token; see [Video generation](#video-generation). `VideoClient` is its Effect service and layer.
10471114
- **`Media`** — the shared asset type (`Media.Asset`, `Media.Source`) and constructors used by messages, tool results, and media requests.
10481115
- **`Generation`** — provider-neutral handle for an in-flight media generation (`await`, `refresh`, `cancel`, `events`) used by queued media routes.
10491116
- **`ai.decision.request` / `ai.decision.generate`** — typed predicate, choice, and score questions over text, JSON, and media input; `DecisionClient` is its Effect service and layer.
10501117
- **`ai.embedding.request` / `ai.embedding.generate`** — one vector per text or media value, with optional `dimensions` and `inputType`; `EmbeddingClient` is its Effect service and layer.
10511118
- **`AIClient.layer` / `AIClient.layerWith(executor)`** — every modality client plus the request executor in one layer.
1052-
- **`@opencode/ai`** (Promise root) — a default `ai` with the text calls (`request`, `generate`, `stream`, `loop`, `compact`, `canCompact`) at the top level plus `image`, `decision` and `embedding` (plus `experimental.video`, `experimental.speech`, and `experimental.transcription`) as Promise / `AsyncIterable` APIs, `run(effect)` for any package Effect, plus `file`, `write`, `bytes`, `base64`, and `materialize` for assets. `AI.make({ layer? })` builds a client over a custom executor. The same `ai` from `@opencode/ai/effect` returns `Effect` and `Stream`.
1053-
- **`@opencode/ai/experimental`** — APIs that may change without compatibility guarantees: `Video`, `Speech`, `Transcription`, `Extraction`, and `Realtime`. See [Experimental](#experimental).
1119+
- **`@opencode/ai`** (Promise root) — a default `ai` with the text calls (`request`, `generate`, `stream`, `loop`, `compact`, `canCompact`) at the top level plus `image`, `decision`, `embedding` and `video` (plus `experimental.speech` and `experimental.transcription`) as Promise / `AsyncIterable` APIs, `run(effect)` for any package Effect, plus `file`, `write`, `bytes`, `base64`, and `materialize` for assets. `AI.make({ layer? })` builds a client over a custom executor. The same `ai` from `@opencode/ai/effect` returns `Effect` and `Stream`.
1120+
- **`@opencode/ai/experimental`** — APIs that may change without compatibility guarantees: `Speech`, `Transcription`, `Extraction`, and `Realtime`. See [Experimental](#experimental).
10541121

10551122
## Experimental
10561123

10571124
These APIs may change without compatibility guarantees. The Effect API is exported from `@opencode/ai/experimental`,
10581125
and the Promise client exposes it under `ai.experimental`.
10591126

1060-
- **`Video`**, **`Speech`**, and **`Transcription`** — media generation and speech-to-text, with `VideoClient`,
1061-
`SpeechClient`, and `TranscriptionClient` as their Effect services. Promise: `ai.experimental.video`, `.speech`, and
1062-
`.transcription`.
1127+
- **`Speech`** and **`Transcription`** — speech generation and speech-to-text, with `SpeechClient` and
1128+
`TranscriptionClient` as their Effect services. Promise: `ai.experimental.speech` and `.transcription`.
10631129
- **`Extraction`** — typed values read out of text.
10641130
- **`Realtime`** — `Realtime.connect` starts an OpenAI, xAI, or Gemini voice session and returns a WebRTC answer SDP or
10651131
a short-lived WebSocket URL and token.
10661132

1067-
The provider selectors for these modalities (`.video(...)`, `.speech(...)`, `.transcription(...)`,
1133+
The provider selectors for these modalities (`.speech(...)`, `.transcription(...)`,
10681134
`.experimental.extraction(...)`) and their protocol entrypoints (such as `@opencode/ai/protocols/openai-speech`) are
10691135
part of the same experimental surface.
10701136

1071-
### Video generation
1072-
1073-
Video mirrors `Image` with one difference: every provider is asynchronous, so the route is a submit-then-poll
1074-
`Generation`. Models come from `.video(...)` selectors on the `XAI` facade.
1075-
Common fields (`frames`, `references`, `video`, `durationSeconds`, `aspect`, `resolution`, `audio`, `n`, `seed`,
1076-
`negativePrompt`) lower natively or fail with a typed `AIError` before any network call; provider-native controls live
1077-
under `providerOptions`, inferred from the selected model.
1078-
1079-
```ts
1080-
import { Video } from "@opencode/ai/experimental"
1081-
import { XAI } from "@opencode/ai/providers"
1082-
1083-
const xai = XAI.configure({ apiKey: process.env.XAI_API_KEY })
1084-
1085-
// Simple: submit and wait.
1086-
const program = Effect.gen(function* () {
1087-
const response = yield* Video.generate(
1088-
{
1089-
model: xai.video("grok-imagine-video-1.5"),
1090-
prompt: "Panning wide shot of a calico kitten sleeping in the sunshine",
1091-
aspect: "16:9",
1092-
resolution: "720p",
1093-
durationSeconds: 10,
1094-
},
1095-
{ poll: { interval: "10 seconds", timeout: "10 minutes" } },
1096-
)
1097-
// xAI returns a temporary URL: materialize before persisting, or the persisted URL may stop working.
1098-
return yield* response.video.materialize()
1099-
})
1100-
1101-
// Explicit control: keep the handle, persist the token, resume elsewhere.
1102-
const controlled = Effect.gen(function* () {
1103-
const generation = yield* Video.start({ model: xai.video("grok-imagine-video-1.5"), prompt })
1104-
generation.id // provider request id
1105-
generation.status // "queued" | "running" | "completed" | "failed" | "cancelled" | "expired"
1106-
generation.token // route-owned JSON, `{ requestID }` for xAI
1107-
// The token carries no route identity: persist the provider and model ID alongside it, since `resume` needs the model.
1108-
const saved = JSON.stringify(generation.token)
1109-
1110-
const resumed = yield* Video.resume(xai.video("grok-imagine-video-1.5"), JSON.parse(saved))
1111-
return yield* resumed.await({ poll: { interval: "10 seconds" } })
1112-
})
1113-
1114-
// Progress as a stream: generation-queued | generation-progress | video | finish.
1115-
const events = Video.stream({ model: xai.video("grok-imagine-video-1.5"), prompt }, { poll })
1116-
```
1117-
1118-
Status polls, result fetches, and cancels all run through the same request executor with the route's
1119-
auth. `Generation.await` and `Generation.events` fail with a
1120-
`Timeout` reason when `poll.timeout` (default 10 minutes) elapses. Status polls and result fetches retry transient
1121-
failures (rate limits, provider 5xx, network errors) with backoff that honors `retry-after`, always within
1122-
`poll.timeout`; submits and cancels never retry. Interrupting a wait (or aborting its `signal`) does not cancel the
1123-
provider job, which keeps running and billing: call `cancel()` to stop it on providers with a cancel endpoint (xAI has
1124-
none, so `cancel()` sends nothing there). Failed, cancelled, and expired generations fail typed with the provider's
1125-
terminal document on `reason.body`; moderation outcomes (xAI `respect_moderation`) fail with a `ContentPolicy` reason
1126-
when no video comes back; when one still does, the raw outcome stays in `providerMetadata`.
1127-
1128-
Provider notes:
1129-
1130-
- **xAI** sends a `video` input to `/videos/edits`, or `/videos/extensions` with `providerOptions.mode: "extend"`.
1131-
`seed` and `negativePrompt` are not supported.
1132-
1133-
The promise client exposes the same surface: `ai.experimental.video.start(...)` resolves to a handle with `await`, `events`,
1134-
`result`, `refresh`, `cancel`, and `token`; `ai.experimental.video.generate`, `ai.experimental.video.resume(model, token)`, and
1135-
`ai.experimental.video.stream` mirror the Effect API. The handle's `status` and `progress` are a snapshot from when it was
1136-
created; `refresh()` resolves to a new handle. Every promise method and stream accepts `{ signal }`: like `fetch`,
1137-
aborting rejects the Promise or throws from the `for await` loop with `signal.reason` (an `AbortError` `DOMException`
1138-
unless `abort(reason)` passed one), while `break` stops a stream without throwing.
1139-
1140-
```ts
1141-
import { ai } from "@opencode/ai"
1142-
1143-
const generation = await ai.experimental.video.start({ model, prompt })
1144-
for await (const event of generation.events({ poll: { interval: 10_000 } })) console.log(event.type)
1145-
const video = await generation.result({ signal })
1146-
await ai.write(video.video, "./kite.mp4")
1147-
```
1148-
11491137
### Speech generation
11501138

11511139
Speech (text-to-speech) is one request whose response is parsed incrementally, so every route supports both

‎packages/ai/src/effect.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,17 +18,18 @@ export const ai = {
1818
image: Image,
1919
decision: Decision,
2020
embedding: Embedding,
21+
video: Video,
2122
/** Experimental: may change without compatibility guarantees. */
2223
experimental: {
2324
speech: Speech,
2425
transcription: Transcription,
25-
video: Video,
2626
},
2727
} as const
2828

2929
export { AIClient } from "./ai-client.js"
3030
export { LLMClient } from "./route/client.js"
3131
export { ImageClient } from "./image-client.js"
32+
export { VideoClient } from "./video-client.js"
3233
export { Auth } from "./route/auth.js"
3334
export { ProviderPackage } from "./provider-package.js"
3435
export type { LLMClientService } from "./route/client.js"

‎packages/ai/src/experimental.ts‎

Lines changed: 0 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,3 @@
1-
export { VideoClient } from "./video-client.js"
2-
export { VideoAspectRatio, VideoEvent, VideoFrames, VideoModel, VideoModelSchema, VideoResponse } from "./video.js"
3-
export type {
4-
VideoModelOptions,
5-
VideoOptions,
6-
VideoRequest,
7-
VideoRequestInput,
8-
VideoResolution,
9-
VideoRoute,
10-
} from "./video.js"
11-
export { Video } from "./video.js"
121
export { SpeechClient } from "./speech-client.js"
132
export { SpeechEvent, SpeechModel, SpeechModelSchema, SpeechResponse, SpeechTimestamp, SpeechVoice } from "./speech.js"
143
export type {

‎packages/ai/src/index.ts‎

Lines changed: 27 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -204,20 +204,23 @@ export const make = (options: Options = {}) => {
204204
options,
205205
)
206206
},
207-
/** Write an asset's bytes like `Media.write`, downloading `url` sources through the executor. */
207+
/** Write an asset's bytes like `Media.write`: streamed through a temporary file that replaces `path` on success. */
208208
write: async (asset: Media.Asset, path: string, options?: RunOptions) => {
209-
const { writeFile } = await import("node:fs/promises")
210-
return run(
211-
asset.bytes().pipe(
212-
Effect.flatMap((data) =>
213-
Effect.tryPromise({
214-
try: (signal) => writeFile(path, data, { signal }),
215-
catch: (cause) => fileError(`Failed to write media file ${path}`, cause),
216-
}),
217-
),
218-
),
219-
options,
220-
)
209+
const { open, rename, rm } = await import("node:fs/promises")
210+
const temporary = `${path}.${crypto.randomUUID()}.tmp`
211+
try {
212+
const file = await open(temporary, "w")
213+
try {
214+
for await (const chunk of iterate(asset.stream(), options)) await file.write(chunk)
215+
} finally {
216+
await file.close()
217+
}
218+
await rename(temporary, path)
219+
} catch (cause) {
220+
await rm(temporary, { force: true })
221+
if (cause instanceof AIError || cause === options?.signal?.reason) throw cause
222+
throw fileError(`Failed to write media file ${path}`, cause)
223+
}
221224
},
222225
request: LLM.request,
223226
generate,
@@ -255,6 +258,17 @@ export const make = (options: Options = {}) => {
255258
generate: <const Model extends EmbeddingModel>(input: EmbeddingRequestInput<Model>, options?: RunOptions) =>
256259
run(Embedding.generate(input), options),
257260
},
261+
video: {
262+
request: Video.request,
263+
/** Submit a generation and return its `{ id, token }` at once; nothing polls. */
264+
start: <const Model extends VideoModel>(input: VideoRequestInput<Model> | VideoRequest, options?: RunOptions) =>
265+
run(Video.start(input), options),
266+
/** One status read for a stored token. */
267+
status: (model: VideoModel, token: unknown, options?: RunOptions) => run(Video.status(model, token), options),
268+
/** The finished videos; fails while the job is still running. */
269+
result: (model: VideoModel, token: unknown, options?: RunOptions) => run(Video.result(model, token), options),
270+
cancel: (model: VideoModel, token: unknown, options?: RunOptions) => run(Video.cancel(model, token), options),
271+
},
258272
/** Experimental: may change without compatibility guarantees. */
259273
experimental: {
260274
speech: {
@@ -285,21 +299,6 @@ export const make = (options: Options = {}) => {
285299
resume: (model: TranscriptionModel, token: unknown, options?: RunOptions) =>
286300
run(Transcription.resume(model, token), options).then(handle),
287301
},
288-
video: {
289-
request: Video.request,
290-
start: <const Model extends VideoModel>(input: VideoRequestInput<Model> | VideoRequest, options?: RunOptions) =>
291-
run(Video.start(input), options).then(handle),
292-
generate: <const Model extends VideoModel>(
293-
input: VideoRequestInput<Model> | VideoRequest,
294-
options?: AwaitOptions & RunOptions,
295-
) => run(Video.generate(input, { poll: options?.poll }), options),
296-
resume: (model: VideoModel, token: unknown, options?: RunOptions) =>
297-
run(Video.resume(model, token), options).then(handle),
298-
stream: <const Model extends VideoModel>(
299-
input: VideoRequestInput<Model> | VideoRequest,
300-
options?: AwaitOptions & RunOptions,
301-
) => iterate(Video.stream(input, { poll: options?.poll }), options),
302-
},
303302
},
304303
dispose: () => runtime.dispose(),
305304
}

‎packages/ai/src/media-model.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,10 +84,16 @@ export const composeRoute = <Request extends MediaRoute.MediaRequest, Event, Res
8484
collect: (events: ReadonlyArray<Event>) => Effect.Effect<Response, AIError>,
8585
): MediaRoute.AnyRoute<Request, Event, Response> => {
8686
if (isStreamInput(route)) return MediaRoute.stream({ ...composition(route, input), collect })
87-
if (isQueuedInput(route)) return MediaRoute.queued(composition(route, input))
87+
if (isQueuedInput(route)) return composeQueuedRoute(route, input)
8888
return MediaRoute.inline(composition(route, input))
8989
}
9090

91+
/** Compose a queued protocol route input with one deployment through `MediaRoute.queued`. */
92+
export const composeQueuedRoute = <Request extends MediaRoute.MediaRequest, Response, Token>(
93+
route: MediaModel.QueuedRouteInput<Request, Response, Token>,
94+
input: MediaRoute.ModelInput,
95+
) => MediaRoute.queued(composition(route, input))
96+
9197
const composition = <Request extends MediaRoute.MediaRequest, Protocol>(
9298
route: MediaModel.RouteInput<Request, Protocol>,
9399
input: MediaRoute.ModelInput,

0 commit comments

Comments
 (0)