Pico Harmony (Swift)

Swift package that wraps OpenAI's Harmony Rust library via UniFFI and ships a prebuilt harmony_uniffiFFI.xcframework. It mirrors the Python API shape for rendering/parsing Harmony-formatted conversations and exposes the full encoding surface in Swift.

Installation (SwiftPM)

Add the package to your Package.swift:

dependencies: [
  .package(url: "https://github.com/picoHarmony/PicoHarmony.git", branch: "master"),
],
targets: [
  .target(
    name: "YourApp",
    dependencies: ["Harmony"]
  ),
]

The binary target harmony_uniffiFFI is checked in under Binaries/. If you need to rebuild it (e.g., after updating the Rust submodule), run scripts/build_uniffi.sh with a Rust toolchain installed via rustup and iOS/macOS targets available.

Offline Tokenizer Behavior

HarmonyEncoding now defaults to bundled/offline tokenizer loading. The package ships o200k_base.tiktoken, and on first use it configures Rust to load from bundled assets.

In most apps, no setup call is required:

let enc = try HarmonyEncoding(name: .harmonyGptOss)

If you need a custom tokenizer location, set TIKTOKEN_ENCODINGS_BASE in your process environment before creating HarmonyEncoding.

Quickstart

import Harmony

let enc = try HarmonyEncoding(name: .harmonyGptOss)

// Create a system message with defaults (model identity, reasoning effort, channels, etc.)
var sys = try SystemContent.makeDefault()
sys.modelIdentity = "You are a helpful assistant."

let convo = Conversation(messages: [
  Message(author: Author(role: .system), content: [.system(sys)]),
  Message.user("What is 2 + 2?")
])

// Render to tokens for model input
let tokens = try enc.renderConversationForCompletion(convo, nextTurnRole: .assistant)

// Parse model output back to messages
let parsed = try enc.parseMessagesFromCompletionTokens(completionTokens, role: .assistant)

Adding Tools

Built-in Tools (Browser, Python)

Tool configurations are fetched from Rust to ensure consistency with the canonical implementation:

var sys = try SystemContent.makeDefault()
sys.conversationStartDate = "2025-01-15"
try sys.withBrowserTool()   // Add web browsing capability
try sys.withPythonTool()    // Add Python code execution

let convo = Conversation(messages: [
  Message(author: Author(role: .system), content: [.system(sys)]),
  Message.user("Search for the latest news about Swift.")
])

Custom Function Tools

Define your own tools in the developer message:

var sys = try SystemContent.makeDefault()
try sys.withBrowserTool()

// Define custom function parameters
let weatherParams: JSONValue = .object([
  "type": .string("object"),
  "properties": .object([
    "location": .object([
      "type": .string("string"),
      "description": .string("City and state, e.g. San Francisco, CA")
    ])
  ]),
  "required": .array([.string("location")])
])

var dev = DeveloperContent()
dev.instructions = "Help the user with weather information."
dev.withFunctionTools([
  ToolDescription(name: "get_weather", description: "Get current weather for a location", parameters: weatherParams)
])

let convo = Conversation(messages: [
  Message(author: Author(role: .system), content: [.system(sys)]),
  Message(author: Author(role: .developer), content: [.developer(dev)]),
  Message.user("What's the weather in Tokyo?")
])

MLX-Swift / Local Inference

PicoHarmony is designed for use with local LLMs via MLX-Swift or similar frameworks. Typical workflow:

import Harmony

let enc = try HarmonyEncoding(name: .harmonyGptOss)

// 1. Build your conversation
var sys = try SystemContent.makeDefault()
let convo = Conversation(messages: [
  Message(author: Author(role: .system), content: [.system(sys)]),
  Message.user("Explain quantum computing briefly.")
])

// 2. Render to token IDs for model input
let inputTokens = try enc.renderConversationForCompletion(convo, nextTurnRole: .assistant)

// 3. Run inference with your model (MLX-Swift, llama.cpp, etc.)
let outputTokens: [UInt32] = model.generate(inputTokens, maxTokens: 512)

// 4. Parse the completion tokens back to messages
let messages = try enc.parseMessagesFromCompletionTokens(outputTokens, role: .assistant)

// Messages may include:
// - "analysis" channel: model's reasoning/thinking
// - "final" channel: the response shown to the user
// - Tool calls with recipient like "functions.get_weather"
for msg in messages {
  print("Channel: \(msg.channel ?? "none"), Content: \(msg.content)")
}

Streaming Responses

For real-time token-by-token parsing during generation:

let parser = try StreamableParser(encoding: enc, role: .assistant)

// Feed tokens as they're generated
for token in outputTokens {
  let delta = try await parser.process(token)
  
  // delta.channel - current channel (analysis, commentary, final)
  // delta.delta - new text content
  // delta.recipient - tool being called (e.g., "functions.get_weather")
  
  if let text = delta.delta {
    print(text, terminator: "") // Stream to UI
  }
}

// Get all parsed messages
let messages = try await parser.messages()

Stop Tokens

Get the appropriate stop tokens for generation:

let stopTokens = try enc.stopTokens()
let actionStopTokens = try enc.stopTokensForAssistantActions() // For tool calls

QoS Priority Inversion Warning

When calling StreamableParser from high-QoS Swift code (e.g., UI-related tasks), you may see this warning in the console:

Thread running at User-initiated quality-of-service class waiting on a
lower QoS thread running at Default quality-of-service class

This warning is benign. It occurs because the underlying Rust parser uses std::sync::Mutex, which doesn't participate in Darwin's QoS priority inheritance. There's no crash risk, no App Store rejection concern, and the OS mitigates via priority boosting.

To suppress the warning, wrap heavy parsing operations in a background task:

// Option 1: Detached task with background priority
Task.detached(priority: .background) {
    for token in outputTokens {
        let delta = try await parser.process(token)
        // Handle delta...
    }
}

// Option 2: Dedicated dispatch queue
let parserQueue = DispatchQueue(label: "com.myapp.parser", qos: .utility)
parserQueue.async {
    // Parsing work here
}

API Parity with Python

This library mirrors the Python openai-harmony API, making it easy to port examples:

Python	Swift
SystemContent.new()	SystemContent.makeDefault()
.with_browser_tool()	.withBrowserTool()
.with_python_tool()	.withPythonTool()
.with_function_tools([...])	.withFunctionTools([...])
load_harmony_encoding(name)	HarmonyEncoding(name:)
StreamableParser(enc, role)	StreamableParser(encoding:role:)

Project Layout

• Sources/Harmony/ – Swift API surface
• rust/harmony_uniffi/ – UniFFI bridge code
• rust/openai-harmony/ – upstream Harmony Rust submodule
• Binaries/harmony_uniffiFFI.xcframework – prebuilt XCFramework (static framework-bundle slices)
• Tests/PicoHarmonyTests/ – Swift test suite (parity with Python fixtures)

Development

# Run tests
swift test

# Rebuild XCFramework (if Rust sources change)
./scripts/build_uniffi.sh

You normally don't need a local Rust toolchain. After changing anything under rust/** (including bumping the rust/openai-harmony submodule) or scripts/build_uniffi.sh, manually run the Build XCFramework GitHub Actions workflow (Actions → Build XCFramework → Run workflow, targeting your branch). It rebuilds Binaries/harmony_uniffiFFI.xcframework on a macOS runner, verifies swift build / swift test against the fresh binaries, and commits them back to that branch.

The rebuild is intentionally manual — run it before merging a Rust or submodule change, otherwise the committed binaries and generated Swift bindings can lag behind the sources (CI links the committed XCFramework and does not rebuild the FFI). A scheduled Check upstream harmony workflow also warns, via a tracking issue, when the pinned openai-harmony submodule falls behind the upstream OpenAI harmony release.

Binary packaging

The XCFramework uses framework-bundle slices (harmony_uniffiFFI.framework with Modules/module.modulemap) instead of bare static-library slices. Bare-library slices put module.modulemap at the root of Headers/, which Xcode stages into the shared Build/Products/<config>/include/ directory — so any build that also links another static-lib XCFramework packaged the same way (e.g. chroma-swift's chroma_swiftFFI) fails with Multiple commands produce '…/include/module.modulemap'. With framework bundles the module map stays inside the uniquely named bundle and no shared path is claimed. The framework binary is still a static archive, so consumers link exactly as before and nothing is embedded at runtime.

For details on the Harmony format itself, see the upstream project: https://github.com/openai/harmony.