QwenAudio Toolkits

QwenAudio Toolkits is a local-first desktop workspace for audio AI creation and model execution. It offers a Creative Workshop for direct, guided tasks such as video editing, AI podcast, video dubbing, and meeting notes, alongside an Agent workspace for open-ended requests. Required models are installed only when they are needed.

The product goal is to become a conversational Agent-driven audio and video creation IDE. The Agent understands user intent, plans workflows, orchestrates models, and produces reviewable edits; dedicated editing surfaces provide timeline, waveform, caption, dubbing, and preview controls; the model store supplies local and cloud capabilities on demand.

Four product pillars guide this goal:

• Creative Workshop workflows: choose a frequent creative goal, set a few fixed parameters, and enter its dedicated workspace without first configuring or conversing with an Agent.

• Conversational Agent workflow: describe an audio or video goal in natural language, then let the Agent plan the workflow, choose tools and models, run long tasks, and return reviewable edits instead of opaque results.

• On-demand open source model store: install open-source local models, runtime packages, dependencies, and cloud model definitions only when a skill or project needs them.

• Dedicated professional editing surfaces: use purpose-built timeline, waveform, transcript, caption, dubbing, and preview interfaces for precise manual control, while the Agent operates the same project state.

The first public preview targets Apple Silicon Macs running macOS 14.2 or later. Model weights and runtime packages are downloaded on demand, so they are not bundled into the application installer.

What it supports

• Speech recognition, voice activity detection, and language-aware audio workflows
• Audio enhancement and noise suppression
• Text normalization, including TN / ITN processing
• Text-to-speech and reference-voice workflows where supported by the model
• Guided video and audio workflows for video editing, captions, dubbing, podcast generation, and meeting notes
• Local model runtimes and cloud API models behind one typed Harness contract
• A model store with variants, checksums, dependencies, and resumable downloads
• Shared input, streaming, preview, and result-detail interactions across model capabilities

The bundled catalog currently focuses on local VAD, ASR, enhancement, TTS, and text-normalization models. Model entries are data-driven and can be refreshed from the project's ModelScope repository.

Download and install

Application binaries are published through GitHub Releases. Download the latest Apple Silicon .dmg, open it, and drag QwenAudio Toolkits into Applications. The app downloads model weights and runtime packages separately from ModelScope after you install a model in the app.

Current preview builds use ad-hoc macOS signing and are not Apple-notarized. If macOS blocks the first launch, open System Settings → Privacy & Security and approve the application. Only download installers from the official project release page.

Run from source

Prerequisites

• Apple Silicon macOS 14.2 or later
• Node.js 20.19 or later and npm 10 or later
• A current stable Rust toolchain
• CMake and a C/C++ compiler
• Xcode Command Line Tools
• Tauri 2 platform prerequisites

git clone https://github.com/QwenAudio/qwen-audio-toolkits.git
cd qwen-audio-toolkits
npm ci
npm run desktop:dev

npm run dev starts a browser-only frontend preview. Native model runtimes, microphone access, system-audio capture, and the updater require the Tauri desktop process.

The local development server can inspect installed ACP Agents and load their model choices. This performs an ACP handshake without sending a prompt or enabling tools. The inspection endpoint is restricted to same-origin loopback requests and is not included in production builds. Agent conversations still run through the desktop process.

For a local production-style build:

npm run desktop:build

This creates a DMG without updater artifacts, so contributors do not need the project's private updater signing key. Maintainers use npm run desktop:release inside the release workflow to create signed updater artifacts.

See the getting started guide for permissions, model installation, cloud configuration, and local data locations.

Start a creation

Use 创意工坊 / Creative Workshop when the goal matches a common, guided workflow. It is the primary entry point for users who want to create directly:

1. Choose a video or audio workflow.
2. Add the required source material, if any.
3. Set the workflow's fixed options, such as caption generation, dubbing language, or meeting-note focus.
4. Start in the dedicated editor. The editor checks and helps install the models it needs; it does not require an Agent conversation or Agent model configuration.

The initial workshop workflows are Smart Cut, Captioned Video, Video Dubbing, AI Podcast, and Meeting Notes. They are grouped into video and audio creation and keep their projects in the regular recent-task history.

Use New Task when the request is open-ended, needs planning, or benefits from natural-language iteration:

1. Open New Task.
2. Describe what you want to create.
3. Optionally select an Agent capability and describe additional constraints.
4. Add source material when the selected skill needs a file.
5. Install or configure the required models if the skill asks for them.

The product distinguishes a user-facing workflow from a reusable Skill. Creative Workshop workflows are fixed task recipes; Skills are declarative, installable capabilities that an Agent or workflow can discover and compose. For example, a video-dubbing workflow can combine transcription, dialogue translation, voice synthesis, and subtitle rendering Skills. The current catalog still exposes some built-in workspace workflows as legacy Skills while this separation is migrated. See Creative Workshop and Skills design.

Agent-created tasks keep the conversation beside their manual editor. Ask AI to change editing parameters, mark cuts, edit podcast turns, adjust dubbing settings, or inspect meeting notes. Creative Workshop tasks open directly in the dedicated editor instead. Both paths operate on the same saved project state. If you edit an Agent workspace while it is planning, its pending changes are stopped so you can send an updated request. Changed podcast or dubbing settings require new audio before the updated result can be exported.

The suggested exact commands, such as 关闭字幕 and 配音风格设为轻松, work without a language model. Other requests use the selected ACP Agent, such as Codex or Qoder. Choose the Agent and its advertised model beside the Send button; Agents that do not advertise models use their own default. Provider selection and model selection are saved with each task. Native generation and recording still require the desktop app and the relevant models.

Use a model directly

1. Open Model Store.
2. Choose an Offline model, a bundled cloud model, or configure a custom REST LLM, ASR, or TTS model.
3. Select a model variant when available and start the installation.
4. Open the installed model from the sidebar.
5. Upload or drag in audio, record from the microphone, or enter text according to the selected capability.

Local weights are downloaded only after installation. Interrupted downloads can be paused, resumed, or canceled. Recommended dependencies, such as VAD or reference transcription, remain separate models and can be selected from the model details.

Cloud execution sends the selected input to the configured provider. Configure provider credentials from the API model configuration flow in Model Store; local models continue to run without access to those credentials.

Updates, models, and privacy

Application updates and model assets use separate channels:

• GitHub Releases: desktop application installers and Tauri updater assets
• ModelScope: model catalog, model weights, and runtime packages

The application checks the GitHub updater manifest in the background and downloads a signed update automatically when one is available. The app asks you to restart before installing the downloaded update. On macOS, you can also choose QwenAudio Toolkits → 检查更新… from the application menu at the upper-left of the screen. Updating the application keeps installed models and application data.

Local inference does not upload audio. Cloud models send the requested input to their configured provider. The app does not include telemetry, advertising analytics, or automatic crash reporting. Application data on macOS is stored under:

~/Library/Application Support/org.qwenaudio.toolkits/

This directory contains installed plugins and model assets, generated and processed audio, recordings, run history, and provider configuration. Removing the app does not remove this directory automatically. See PRIVACY.md for the complete storage and network boundaries.

Task conversations, drafts, and the four creation editors save automatically to workspace/workspace-v1.json in this directory. Reopening the app restores the selected task and saved edits; interrupted processing waits for an explicit retry, and meeting recording stays stopped. The header shows save progress or an error, and normal desktop closing waits for pending writes.

Snapshots keep media file references, not copies of source media or live meeting audio. Keep those files in place; missing files leave the saved text and edits available. Podcast audio is tied to its script and voice settings: editing them requires an audio update before export, while unchanged speech segments can be reused. Browser development uses local storage for the current origin; ?demo does not read or overwrite saved tasks. An unreadable or unsupported workspace file is preserved and autosaving is paused. Back it up before repairing or moving it aside, then restart the app.

Creative Workshop, Skills, and model projects

The desktop UI separates Creative Workshop from Skills and the Model Store. Creative Workshop owns user-facing task recipes; Skills own reusable capability definitions, instructions, contracts, and requirements; Model Store entries provide concrete local or cloud model implementations. During the current migration period, imported skill projects and model projects use the existing manifest contracts. See Creative Workshop and Skills design for the target boundary and Skill and model projects for the current host-adapter execution boundary.

Architecture

React / TypeScript workspace
            │ Tauri commands + events
            ▼
Rust Harness runtime ─── local HTTP API (127.0.0.1:3847)
            │
            ├── reviewed local adapters ── on-demand model assets
            └── configured cloud providers

The Harness exposes a finite set of capabilities, ports, and parameter types. Model plugins cannot inject arbitrary native code into the main process. New runtime architectures require a reviewed adapter; compatible models can then reuse it through declarative manifests.

• Architecture
• Model catalog
• Model Plugin v2
• Plugin manifest schema

Development and validation

npm run lint
npm test
npm run build
npm run open-source:check
cargo fmt --manifest-path src-tauri/Cargo.toml --check
cargo test --manifest-path src-tauri/Cargo.toml

The local model smoke test requires a running desktop app and installed test models:

npm run models:smoke

Please read CONTRIBUTING.md before making a substantial change. Report security issues privately according to SECURITY.md.

Release for maintainers

The release workflow is defined in .github/workflows/release.yml. It builds an Apple Silicon DMG and creates a draft GitHub Release containing the installer, signed updater artifacts, and latest.json.

Before the first release, configure these repository Actions secrets:

• TAURI_SIGNING_PRIVATE_KEY
• TAURI_SIGNING_PRIVATE_KEY_PASSWORD when the key is password-protected

Never commit or share the private key. Then update the version, run the checks, and start Release desktop app from the Actions tab with a matching tag such as v0.1.0. The workflow verifies that the tag matches package.json, so the version and tag must be identical.

The complete process is documented in docs/releasing.md.

License

The original project source is licensed under the Apache License 2.0. Third-party runtimes, libraries, model weights, datasets, and hosted services retain their own licenses and terms. See NOTICE and THIRD_PARTY_NOTICES.md.