diff --git a/CHANGELOG.md b/CHANGELOG.md index cf235767..9ff21473 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,27 @@ The format is based on Keep a Changelog. ### Runtime +- added `vision-chat-q38-27b`, pinned to Qwen's official Apache-2.0 + Qwen3.8-27B BF16 checkpoint. The native Qwen-family runtime now exposes the + dense 27B hybrid-attention model's text, code, and image understanding, + published 262,144-token context, thinking and sampling defaults, complete + generation EOS set, and Qwen3.8 image sizing. The 55.59 GB pull is explicit + rather than an inference-time auto-download and is gated to 64 GB unified + memory with 96 GB recommended. The coding benchmark accepts Qwen3.8 as an + explicit model and scores only visible code after separating reasoning. The + runtime can load the checkpoint's embedded dense MTP head for opt-in + three-token greedy drafts. It remains experimental because BF16 multi-token + verification can diverge from serial greedy decode; the default, sampled, + and constrained requests retain the target-only path. +- added the separate `vision-chat-q38-27b-4bit` managed lane, pinned to LM + Studio's 4-bit/group-64 MLX conversion. Its 19.47 GB composite pull mounts + only Qwen's pinned final BF16 shard under `mtp/`, retaining the official MTP + head and license without duplicating the 55.59 GB checkpoint. Target-only + decode remains the default; `MERERUN_Q35_MTP_SPECULATION=1` enables the + explicitly experimental fast path. A deterministic 24-task stride through + the official HumanEval set scored 20/24 in both modes with the same failures; + one failing case changed length by three tokens, so MTP remains opt-in rather + than claiming exact greedy-output parity. - refreshed the owned MLX and mlx-swift forks onto current upstream cutoffs, retained the scoped 1-bit, NVFP4, CUDA, stream-safety, and M4/H3 patches, made the generated NAX optimizations source-reproducible, and regenerated diff --git a/README.md b/README.md index 7bdaea92..ee4219b2 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ current flags. | Area | Public commands | Current surface | | --- | --- | --- | | Images and LoRAs | `image generate`, `image train-lora`, `adapter list`, `adapter pull` | Klein, ZImage, HiDream O1, Krea 2, Ideogram 4, and Bonsai; text-to-image, edits, multiple references, structured prompts, local Krea/Klein training, and checksum-pinned public adapters | -| Text, code, and agents | `text chat`, `text code`, `text embed`, `text anonymize`, `text train-lora`, `agent` | Local chat and tool use, including Bonsai 27B binary/ternary vision chat; code generation, embeddings, PII redaction, text LoRA training, and guided local-agent setup | +| Text, code, and agents | `text chat`, `text code`, `text embed`, `text anonymize`, `text train-lora`, `agent` | Local chat and tool use, including Qwen3.8 27B BF16/4-bit and Bonsai 27B binary/ternary vision chat; code generation, embeddings, PII redaction, text LoRA training, and guided local-agent setup | | Vision understanding | `vision caption`, `inspect`, `face`, `ground`, `segment`, `track`, `track-live`, `pose`, `flow`, `ocr` | Captioning and VQA, local face detection/identity embeddings, LightOn/GLM/Infinity OCR, Falcon grounding, SAM 3.1 segmentation and tracking, body/hand/face landmarks, and dense optical flow | | Depth, geometry, and 3D | `vision depth-video`, `geometry`, `geometry-multiview`, `image-to-3d*`; `image reconstruct-3d*` | Video Depth Anything, MoGe-2, Depth Anything 3, TripoSR, InstantMesh, and TRELLIS.2; depth/confidence EXRs, cameras, point clouds, 3DGS initialization, OBJ, PLY, GLB, and PBR voxel artifacts | | Audio enhancement | `audio enhance` | Native AP-BWE speech bandwidth extension and UniverSR general-audio super-resolution to hashed 48 kHz mono WAVs | diff --git a/Sources/MereRunCLI/Commands/ModelBenchmarkCodeCommand.swift b/Sources/MereRunCLI/Commands/ModelBenchmarkCodeCommand.swift index c0e675c9..3aba9c96 100644 --- a/Sources/MereRunCLI/Commands/ModelBenchmarkCodeCommand.swift +++ b/Sources/MereRunCLI/Commands/ModelBenchmarkCodeCommand.swift @@ -257,7 +257,7 @@ struct ModelBenchmarkCode: AsyncParsableCommand { guard let spec = ManagedModelCatalog.spec(for: modelID) else { return CodeBenchmarkModelResult.missing(model: modelID, reason: "Unknown model id.") } - guard spec.category == .textCode else { + guard Self.supportsCodingBenchmark(spec) else { return CodeBenchmarkModelResult.missing(model: modelID, reason: "Model is not a text-code model.") } guard let installedURL = ManagedModelResolver.resolveInstalledModel(id: modelID) else { @@ -344,7 +344,7 @@ struct ModelBenchmarkCode: AsyncParsableCommand { do { let response = try await generate(request) let generationSeconds = Date().timeIntervalSince(generationStart) - let candidate = task.candidateProgram(from: response.response) + let candidate = task.candidateProgram(from: Self.scoredCodeResponse(response)) let execution = try CodeExecutionSandbox.runPython( program: task.testProgram(candidateProgram: candidate), python: python, @@ -422,6 +422,15 @@ struct ModelBenchmarkCode: AsyncParsableCommand { || ManagedModelCatalog.spec(for: modelID)?.validationKind == .q35 } + static func supportsCodingBenchmark(_ spec: ManagedModelSpec) -> Bool { + spec.category == .textCode + || Q35Resources.isQ38ModelId(spec.id) + } + + static func scoredCodeResponse(_ response: ChatResponse) -> String { + ChatReasoningMarkup.splitThinkBlocks(in: response.response).visibleContent + } + static let systemPrompt = """ You are completing Python programming benchmark tasks. Return only valid Python code. Do not include Markdown fences, prose, comments about your approach, or test code. diff --git a/Sources/MereRunCore/ManagedModelCatalog.swift b/Sources/MereRunCore/ManagedModelCatalog.swift index be5b32f6..00948d29 100644 --- a/Sources/MereRunCore/ManagedModelCatalog.swift +++ b/Sources/MereRunCore/ManagedModelCatalog.swift @@ -1652,6 +1652,56 @@ public enum ManagedModelCatalog { defaultCLICommands: ["chat", "api serve"], apiProfile: .q36(contextWindow: Q35Resources.defaultContextLength) ), + ManagedModelSpec( + id: Q35Resources.q38TwentySevenBModelId, + category: .visionChat, + installShape: .directoryRoot, + hubFallback: Q35Resources.profile(for: Q35Resources.q38TwentySevenBModelId)?.hubFallbackConfig, + upstreamRepoId: Q35Resources.q38TwentySevenBUpstreamRepoId, + upstreamRevision: Q35Resources.q38TwentySevenBUpstreamRevision, + validationKind: .q35, + runtimeAutoDownloadAllowed: false, + estimatedDownloadBytes: Q35Resources.q38TwentySevenBEstimatedDownloadBytes, + defaultCLICommands: [ + "text chat", + "api serve", + "model benchmark chat", + "model benchmark code", + "model benchmark vlm", + ], + apiProfile: .q36(contextWindow: Q35Resources.q38TwentySevenBContextLength) + ), + ManagedModelSpec( + id: Q35Resources.q38TwentySevenB4BitModelId, + category: .visionChat, + installShape: .directoryRoot, + hubFallback: Q35Resources.profile( + for: Q35Resources.q38TwentySevenB4BitModelId + )?.hubFallbackConfig, + mountedHubFallbacks: [ + MountedHubFallbackConfig( + destinationPath: Q35Resources.q38MTPComponentPath, + hubFallback: HubFallbackConfig( + repoId: Q35Resources.q38TwentySevenBUpstreamRepoId, + revision: Q35Resources.q38TwentySevenBUpstreamRevision, + patterns: Q35Resources.q38MTPComponentSnapshotPatterns + ) + ), + ], + upstreamRepoId: Q35Resources.q38TwentySevenB4BitUpstreamRepoId, + upstreamRevision: Q35Resources.q38TwentySevenB4BitUpstreamRevision, + validationKind: .q35, + runtimeAutoDownloadAllowed: false, + estimatedDownloadBytes: Q35Resources.q38TwentySevenB4BitEstimatedDownloadBytes, + defaultCLICommands: [ + "text chat", + "api serve", + "model benchmark chat", + "model benchmark code", + "model benchmark vlm", + ], + apiProfile: .q36(contextWindow: Q35Resources.q38TwentySevenBContextLength) + ), ManagedModelSpec( id: Q35Resources.bonsai27B1BitModelId, category: .textChat, @@ -3513,7 +3563,12 @@ public extension ManagedModelSpec { case .lagunaDFlash: return LagunaResources.missingDFlashFiles(rootURL: rootURL, fileManager: fileManager) case .q35: - return Q35Resources(rootURL: rootURL).validate(fileManager: fileManager) + let resources = Q35Resources(rootURL: rootURL) + var missing = resources.validate(fileManager: fileManager) + if id == Q35Resources.q38TwentySevenB4BitModelId { + missing.append(contentsOf: resources.validateQ38MTPComponent(fileManager: fileManager)) + } + return missing case .lfm2: return LFM2Resources(rootURL: rootURL).validate( fileManager: fileManager, diff --git a/Sources/MereRunCore/ManagedModelSupport.swift b/Sources/MereRunCore/ManagedModelSupport.swift index 65080946..81cae70e 100644 --- a/Sources/MereRunCore/ManagedModelSupport.swift +++ b/Sources/MereRunCore/ManagedModelSupport.swift @@ -409,6 +409,20 @@ public enum ManagedModelCapabilityCatalog { recommended: 32, setup: true ), + descriptor( + Q35Resources.q38TwentySevenBModelId, + "Qwen3.8 27B vision chat", + "Runs Qwen's official dense 27B BF16 vision-language checkpoint through the native Qwen-family runtime.", + minimum: 64, + recommended: 96 + ), + descriptor( + Q35Resources.q38TwentySevenB4BitModelId, + "Qwen3.8 27B 4-bit vision chat", + "Runs LM Studio's pinned MLX 4-bit conversion with Qwen's official MTP shard available for explicit speculative decode.", + minimum: 32, + recommended: 48 + ), descriptor( Q35Resources.bonsai27B1BitModelId, "Bonsai 27B 1-bit vision chat", diff --git a/Sources/MereRunCore/MereRunModelManifest.swift b/Sources/MereRunCore/MereRunModelManifest.swift index 4dcc7ff9..e137395d 100644 --- a/Sources/MereRunCore/MereRunModelManifest.swift +++ b/Sources/MereRunCore/MereRunModelManifest.swift @@ -30,7 +30,7 @@ public struct MereRunModelManifest: Codable, Hashable, Sendable { case lfm2 = "lfm2" /// Poolside Laguna family via the native Swift runtime. case laguna = "laguna" - /// Q35 family (Qwen3.5 hybrid MoE + hybrid attention). + /// Qwen3.5-family dense or MoE models with hybrid attention. case qwen35HybridMoE = "qwen3.5-hybrid-moe" /// SAM image segmentation family. case samSegmentation = "sam-segmentation" @@ -1086,6 +1086,37 @@ public struct MereRunModelManifest: Codable, Hashable, Sendable { upstreamRepoId: "\(Q35Resources.q36NanoUpstreamRepoId)@\(Q35Resources.q36NanoUpstreamRevision)", createdAt: createdAt ) + case .q38TwentySevenB: + return MereRunModelManifest( + id: modelID.rawValue, + engine: .qwen35HybridMoE, + family: .qwen, + tier: .latest, + variant: .standard, + precision: .bf16, + defaults: nil, + supports: [.chat, .codeGeneration, .visionChat], + components: q35TextComponents, + upstreamRepoId: "\(Q35Resources.q38TwentySevenBUpstreamRepoId)" + + "@\(Q35Resources.q38TwentySevenBUpstreamRevision)", + createdAt: createdAt + ) + case .q38TwentySevenB4Bit: + return MereRunModelManifest( + id: modelID.rawValue, + engine: .qwen35HybridMoE, + family: .qwen, + tier: .latest, + variant: .standard, + precision: .int4, + quantization: Quantization(bits: 4, groupSize: 64, scheme: "mlx-affine"), + defaults: nil, + supports: [.chat, .codeGeneration, .visionChat], + components: q35TextComponents, + upstreamRepoId: "\(Q35Resources.q38TwentySevenB4BitUpstreamRepoId)" + + "@\(Q35Resources.q38TwentySevenB4BitUpstreamRevision)", + createdAt: createdAt + ) case .bonsai27B1Bit: return MereRunModelManifest( id: modelID.rawValue, diff --git a/Sources/MereRunCore/MereRunModelValidator.swift b/Sources/MereRunCore/MereRunModelValidator.swift index b62a293e..03b9f0c3 100644 --- a/Sources/MereRunCore/MereRunModelValidator.swift +++ b/Sources/MereRunCore/MereRunModelValidator.swift @@ -648,7 +648,9 @@ public enum MereRunModelValidator { return .liquid } if modelId == ModelResolver.ModelID.q36Nano.rawValue - || modelId == ModelResolver.ModelID.q36NanoGGUF.rawValue { + || modelId == ModelResolver.ModelID.q36NanoGGUF.rawValue + || modelId == ModelResolver.ModelID.q38TwentySevenB.rawValue + || modelId == ModelResolver.ModelID.q38TwentySevenB4Bit.rawValue { return .qwen } return nil diff --git a/Sources/MereRunCore/ModelResolver.swift b/Sources/MereRunCore/ModelResolver.swift index e50cc7e3..e83be2e5 100644 --- a/Sources/MereRunCore/ModelResolver.swift +++ b/Sources/MereRunCore/ModelResolver.swift @@ -44,6 +44,8 @@ public struct ModelResolver { case nemotron35LightningDSpark = "text-chat-nemotron-35-lightning-dspark" case ltxGemma3TwelveB4Bit = "text-encoder-ltx-gemma3-12b-4bit" case q36Nano = "text-chat-q36-nano" + case q38TwentySevenB = "vision-chat-q38-27b" + case q38TwentySevenB4Bit = "vision-chat-q38-27b-4bit" case bonsai27B1Bit = "text-chat-bonsai-27b-1bit" case bonsai27B2Bit = "text-chat-bonsai-27b-2bit" case lfm25A1B8Bit = "text-chat-lfm25-a1b-8bit" diff --git a/Sources/MereRunCore/Q35/Q35Config.swift b/Sources/MereRunCore/Q35/Q35Config.swift index 46a17b43..0c989e5b 100644 --- a/Sources/MereRunCore/Q35/Q35Config.swift +++ b/Sources/MereRunCore/Q35/Q35Config.swift @@ -1,5 +1,27 @@ import Foundation +public struct Q35GenerationConfig: Codable, Sendable, Hashable { + public let eosTokenIds: [Int] + + private enum CodingKeys: String, CodingKey { + case eosTokenId = "eos_token_id" + } + + public init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: CodingKeys.self) + if let single = try? container.decode(Int.self, forKey: .eosTokenId) { + eosTokenIds = [single] + } else { + eosTokenIds = try container.decodeIfPresent([Int].self, forKey: .eosTokenId) ?? [] + } + } + + public func encode(to encoder: Encoder) throws { + var container = encoder.container(keyedBy: CodingKeys.self) + try container.encode(eosTokenIds, forKey: .eosTokenId) + } +} + public struct Q35QuantizationConfig: Codable, Sendable, Hashable { public let groupSize: Int public let bits: Int diff --git a/Sources/MereRunCore/Q35/Q35Generator.swift b/Sources/MereRunCore/Q35/Q35Generator.swift index 8c99fde4..2a57c439 100644 --- a/Sources/MereRunCore/Q35/Q35Generator.swift +++ b/Sources/MereRunCore/Q35/Q35Generator.swift @@ -151,6 +151,7 @@ public actor Q35Generator: ChatGenerator { private var mtpModel: Q35MTPModel? private var loadedModelPath: String? private var loadedConfig: Q35Config? + private var loadedGenerationEOSTokenIds: [Int] = [] private var loadedResources: Q35Resources? private let modelId: String @@ -179,43 +180,46 @@ public actor Q35Generator: ChatGenerator { modelId: String = Q35Resources.defaultModelId, prefixKVCacheEnabled: Bool = ProcessInfo.processInfo.environment["MERERUN_Q35_PREFIX_KV_CACHE"] == "1", continuousBatchingEnabled: Bool = ProcessInfo.processInfo.environment["MERERUN_Q35_CONTINUOUS_BATCHING"] == "1", - visionMinPixels: Int = Q35Generator.qwen3VLMinPixels, - visionMaxPixels: Int = Q35Generator.qwen3VLMaxPixels + visionMinPixels: Int? = nil, + visionMaxPixels: Int? = nil ) { + let visionPixelBounds = Q35Resources.visionPixelBounds(forModelId: modelId) self.modelId = modelId self.prefixKVCacheEnabled = prefixKVCacheEnabled self.continuousBatchingEnabled = continuousBatchingEnabled - self.visionMinPixels = visionMinPixels - self.visionMaxPixels = visionMaxPixels + self.visionMinPixels = visionMinPixels ?? visionPixelBounds.minimum + self.visionMaxPixels = visionMaxPixels ?? visionPixelBounds.maximum } static func qwen3VLTargetSize( originalWidth width: Int, originalHeight height: Int, patchSize: Int, - temporalPatchSize: Int, spatialMergeSize: Int, minPixels: Int = Q35Generator.qwen3VLMinPixels, maxPixels: Int = Q35Generator.qwen3VLMaxPixels - ) -> (width: Int, height: Int) { + ) throws -> (width: Int, height: Int) { + let aspectRatio = Double(max(width, height)) / Double(min(width, height)) + guard aspectRatio <= 200 else { + throw Q35Error.generationFailed( + "Qwen-family image aspect ratio must not exceed 200; received \(aspectRatio)." + ) + } let factor = max(1, patchSize * max(1, spatialMergeSize)) - let temporalFactor = max(1, temporalPatchSize) - let frames = 1 func roundedToFactor(_ value: Int) -> Int { - max(factor, Int((Double(value) / Double(factor)).rounded()) * factor) + max(factor, Int((Double(value) / Double(factor)).rounded(.toNearestOrEven)) * factor) } var targetHeight = roundedToFactor(height) var targetWidth = roundedToFactor(width) - let temporalPaddedFrames = Int(ceil(Double(frames) / Double(temporalFactor))) * temporalFactor - if temporalPaddedFrames * targetHeight * targetWidth > maxPixels { - let beta = sqrt(Double(frames * height * width) / Double(maxPixels)) + if targetHeight * targetWidth > maxPixels { + let beta = sqrt(Double(height * width) / Double(maxPixels)) targetHeight = max(factor, Int(floor(Double(height) / beta / Double(factor))) * factor) targetWidth = max(factor, Int(floor(Double(width) / beta / Double(factor))) * factor) - } else if temporalPaddedFrames * targetHeight * targetWidth < minPixels { - let beta = sqrt(Double(minPixels) / Double(frames * height * width)) + } else if targetHeight * targetWidth < minPixels { + let beta = sqrt(Double(minPixels) / Double(height * width)) targetHeight = Int(ceil(Double(height) * beta / Double(factor))) * factor targetWidth = Int(ceil(Double(width) * beta / Double(factor))) * factor } @@ -298,6 +302,7 @@ public actor Q35Generator: ChatGenerator { mtpModel = nil loadedModelPath = nil loadedConfig = nil + loadedGenerationEOSTokenIds = [] loadedResources = nil Memory.clearCache() } @@ -342,6 +347,16 @@ public actor Q35Generator: ChatGenerator { progressHandler?(ChatProgress(stage: .loadingModel, message: "Loading Qwen-family config")) let configData = try Data(contentsOf: normalizedRoot.appendingPathComponent("config.json")) let config = try JSONDecoder().decode(Q35Config.self, from: configData) + let generationConfigURL = normalizedRoot.appendingPathComponent("generation_config.json") + let generationEOSTokenIds: [Int] + if FileManager.default.fileExists(atPath: generationConfigURL.path) { + let generationConfigData = try Data(contentsOf: generationConfigURL) + generationEOSTokenIds = try JSONDecoder() + .decode(Q35GenerationConfig.self, from: generationConfigData) + .eosTokenIds + } else { + generationEOSTokenIds = [] + } progressHandler?(ChatProgress(stage: .loadingModel, message: "Loading Qwen-family tokenizer")) let tokenizer = try Q35TokenizerAndTemplate.load( @@ -364,33 +379,27 @@ public actor Q35Generator: ChatGenerator { ) let tower = config.visionConfig == nil ? nil : Q35VisionTower(config: config) - let mtpURL = normalizedRoot.appendingPathComponent("mtp.safetensors") // Load the MTP draft head whenever it ships with the model and isn't // explicitly disabled. Whether speculation is actually USED is decided - // per request by prompt length (see Self.shouldSpeculate): MTP speculative - // decode regresses at short context (measured ~-20-30%) but is a large win - // at long context (~+1.5-2.5x past ~6-8K tokens) on both Metal and CUDA. - let mtpExplicitlyDisabled = { - guard let raw = ProcessInfo.processInfo.environment["MERERUN_Q35_MTP_SPECULATION"]?.lowercased() - else { return false } - return raw == "0" || raw == "false" || raw == "no" - }() + // by the model-specific policy in Self.shouldSpeculate. Dense Qwen3.8 + // benefits at short context; hybrid MoE Qwen retains the measured + // long-context threshold. + let mtpPolicy = ProcessInfo.processInfo.environment["MERERUN_Q35_MTP_SPECULATION"]? + .trimmingCharacters(in: .whitespacesAndNewlines).lowercased() + let mtpExplicitlyDisabled = mtpPolicy == "0" || mtpPolicy == "false" || mtpPolicy == "no" + let mtpExplicitlyEnabled = mtpPolicy == "1" || mtpPolicy == "true" + || mtpPolicy == "yes" || mtpPolicy == "on" + let shouldLoadMTP = !mtpExplicitlyDisabled + && (config.textConfig.usesMoE || mtpExplicitlyEnabled) let loadedMTP: Q35MTPModel? - if !mtpExplicitlyDisabled, FileManager.default.fileExists(atPath: mtpURL.path) { + if shouldLoadMTP, let mtpResources = Self.mtpResources(primary: resources) { progressHandler?(ChatProgress(stage: .loadingModel, message: "Loading Qwen-family MTP weights")) let mtp = Q35MTPModel(config: config) - let arrays = try MLX.loadArrays(url: mtpURL) - try HFSafetensorsWeightsLoader.applyQuantizedWeightsFromArrays( - arrays, - to: mtp, + try loadMTPWeights( + into: mtp, + from: mtpResources, groupSize: groupSize, - bits: bits, - keyMapper: { key in - if key.hasPrefix("mtp.") { - return String(key.dropFirst("mtp.".count)) - } - return "__unused__.\(key)" - } + bits: bits ) loadedMTP = mtp } else { @@ -402,6 +411,7 @@ public actor Q35Generator: ChatGenerator { visionTower = tower mtpModel = loadedMTP loadedConfig = config + loadedGenerationEOSTokenIds = generationEOSTokenIds loadedResources = resources loadedModelPath = normalizedRoot.path } @@ -463,7 +473,11 @@ public actor Q35Generator: ChatGenerator { .write(toFile: dumpPath, atomically: true, encoding: .utf8) } - let eosSet = Set(loadedConfig.eosTokenIds + [tokenizerAndTemplate.eosTokenId].compactMap { $0 }) + let eosSet = Set( + loadedConfig.eosTokenIds + + loadedGenerationEOSTokenIds + + [tokenizerAndTemplate.eosTokenId].compactMap { $0 } + ) let generationConfig = GenerationConfig( maxTokens: request.maxTokens, temperature: Float(request.temperature), @@ -655,17 +669,23 @@ public actor Q35Generator: ChatGenerator { /// Decide whether to use MTP speculative decode for a request. /// - /// Speculative decode only pays off at long context: each main-model pass gets - /// more expensive as the KV cache grows, so verifying several drafted tokens per - /// pass amortizes — but at short prompts the draft-head overhead dominates. - /// Measured (Qwen3.6-35B-A3B OptiQ-4bit, M4 Max): ~20-tok ctx -31%, ~4K -22%, - /// ~12K +1.5-2.5x. Default to speculating only when the prompt and request - /// context are long; MERERUN_Q35_MTP_SPECULATION can enable/disable it, and - /// MERERUN_Q35_MTP_MIN_PROMPT_TOKENS tunes the threshold. - static func shouldSpeculate(promptTokenCount: Int, maxContextTokens: Int? = nil) -> Bool { + /// Select the model-specific MTP break-even point. Qwen3.6 hybrid MoE uses + /// the measured long-context threshold (~20-token context -31%, ~4K -22%, + /// ~12K +1.5-2.5x on M4 Max). Dense Qwen3.8 uses a zero threshold only + /// after explicit opt-in because its BF16 multi-token verification is not + /// serial-greedy identical. MERERUN_Q35_MTP_SPECULATION can enable/disable + /// the path and MERERUN_Q35_MTP_MIN_PROMPT_TOKENS overrides either default. + static func shouldSpeculate( + promptTokenCount: Int, + maxContextTokens: Int? = nil, + defaultMinimumPromptTokens: Int = 6144, + enabledByDefault: Bool = true + ) -> Bool { shouldSpeculate( promptTokenCount: promptTokenCount, maxContextTokens: maxContextTokens, + defaultMinimumPromptTokens: defaultMinimumPromptTokens, + enabledByDefault: enabledByDefault, environment: ProcessInfo.processInfo.environment ) } @@ -673,16 +693,26 @@ public actor Q35Generator: ChatGenerator { static func shouldSpeculate( promptTokenCount: Int, maxContextTokens: Int?, + defaultMinimumPromptTokens: Int = 6144, + enabledByDefault: Bool = true, environment env: [String: String] ) -> Bool { - let threshold = env["MERERUN_Q35_MTP_MIN_PROMPT_TOKENS"].flatMap { Int($0) } ?? 6144 + let rawPolicy = env["MERERUN_Q35_MTP_SPECULATION"]? + .trimmingCharacters(in: .whitespacesAndNewlines).lowercased() + if rawPolicy == "0" || rawPolicy == "false" || rawPolicy == "no" { + return false + } + let explicitlyEnabled = rawPolicy == "1" || rawPolicy == "true" + || rawPolicy == "yes" || rawPolicy == "on" + if !enabledByDefault, !explicitlyEnabled { + return false + } + + let threshold = env["MERERUN_Q35_MTP_MIN_PROMPT_TOKENS"].flatMap { Int($0) } + ?? defaultMinimumPromptTokens let contextAllowsSpeculation = maxContextTokens.map { $0 >= threshold } ?? true - if let raw = env["MERERUN_Q35_MTP_SPECULATION"]?.lowercased() { - if raw == "0" || raw == "false" || raw == "no" { return false } - if raw == "1" || raw == "true" || raw == "yes" || raw == "on" { - return contextAllowsSpeculation - } - // any other value (e.g. "auto") falls through to the adaptive threshold + if explicitlyEnabled { + return contextAllowsSpeculation } if !contextAllowsSpeculation { return false @@ -719,7 +749,9 @@ public actor Q35Generator: ChatGenerator { } let speculationMTP = !jsonConstrained && Self.shouldSpeculate( promptTokenCount: promptTokens.count, - maxContextTokens: maxContextTokens + maxContextTokens: maxContextTokens, + defaultMinimumPromptTokens: model.config.textConfig.usesMoE ? 6144 : 0, + enabledByDefault: model.config.textConfig.usesMoE ) && mropeRopeDelta == nil ? mtpModel : nil let decodePath = Self.decodePath( jsonConstrained: jsonConstrained, @@ -818,6 +850,10 @@ public actor Q35Generator: ChatGenerator { var streamedJSONText = "" var firstTokenSeconds: Double? var jsonGrammar = JSONObjectPrefixGrammar() + var mtpDraftedTokens = 0 + var mtpAcceptedTokens = 0 + var mtpVerificationPasses = 0 + var mtpReplacementPasses = 0 let decodeStart = Date() func emit(_ token: Int) { @@ -918,6 +954,7 @@ public actor Q35Generator: ChatGenerator { guard !draftTokens.isEmpty else { continue } + mtpDraftedTokens += draftTokens.count let candidateCaches = forkLayerCaches(layerCaches) let candidateInput = MLXArray(([next] + draftTokens).map(Int32.init)) @@ -925,6 +962,7 @@ public actor Q35Generator: ChatGenerator { let candidate = model.forward(candidateInput, cache: candidateCaches) MLX.eval(candidate.logits) MLX.eval(candidate.hidden) + mtpVerificationPasses += 1 var accepted = 0 var verificationHistory = repetitionHistory @@ -944,6 +982,7 @@ public actor Q35Generator: ChatGenerator { } if accepted == draftTokens.count { + mtpAcceptedTokens += accepted var hitEOS = false for token in draftTokens { if eosSet.contains(token) { @@ -962,6 +1001,7 @@ public actor Q35Generator: ChatGenerator { } let acceptedPrefix = Array(draftTokens.prefix(accepted)) + mtpAcceptedTokens += accepted var hitEOS = false for token in acceptedPrefix { if eosSet.contains(token) { @@ -987,6 +1027,7 @@ public actor Q35Generator: ChatGenerator { let replacementForward = model.forward(replacementInput, cache: replacementCaches) MLX.eval(replacementForward.logits) MLX.eval(replacementForward.hidden) + mtpReplacementPasses += 1 emit(replacement) layerCaches = replacementCaches logits = lastTokenLogits(replacementForward.logits) @@ -1001,6 +1042,7 @@ public actor Q35Generator: ChatGenerator { baseModel: model ) MLX.eval(draftLogits) + mtpDraftedTokens += 1 let draftProbs = samplingProbabilities( logits: draftLogits[0, -1, 0...], @@ -1014,6 +1056,7 @@ public actor Q35Generator: ChatGenerator { let candidate = model.forward(candidateInput, cache: candidateCaches) MLX.eval(candidate.logits) MLX.eval(candidate.hidden) + mtpVerificationPasses += 1 let targetProbs = samplingProbabilities( logits: candidate.logits[0, 0, 0...], @@ -1025,6 +1068,7 @@ public actor Q35Generator: ChatGenerator { let acceptProbability = min(1.0, targetProb / draftProb) if Float.random(in: 0..<1) <= acceptProbability { + mtpAcceptedTokens += 1 if eosSet.contains(draft) { break } @@ -1049,6 +1093,7 @@ public actor Q35Generator: ChatGenerator { let replacementForward = model.forward(replacementInput, cache: replacementCaches) MLX.eval(replacementForward.logits) MLX.eval(replacementForward.hidden) + mtpReplacementPasses += 1 emit(replacement) layerCaches = replacementCaches logits = lastTokenLogits(replacementForward.logits) @@ -1078,9 +1123,25 @@ public actor Q35Generator: ChatGenerator { } } + let decodeSeconds = Date().timeIntervalSince(decodeStart) + if Gemma4DecodeTrace.enabled, mtpModel != nil { + let acceptance = mtpDraftedTokens > 0 + ? Double(mtpAcceptedTokens) / Double(mtpDraftedTokens) * 100 + : 0 + Gemma4DecodeTrace.emit(String( + format: "[q35-decode-trace] mode=mtp tokens=%d drafted=%d accepted=%d acceptance=%.1f%% verify=%d replacement=%d wall=%.2fms/tok", + generated.count, + mtpDraftedTokens, + mtpAcceptedTokens, + acceptance, + mtpVerificationPasses, + mtpReplacementPasses, + decodeSeconds / Double(max(1, generated.count)) * 1000 + )) + } return Q35BatchedDecodeResult( generatedTokens: generated, - decodeSeconds: Date().timeIntervalSince(decodeStart), + decodeSeconds: decodeSeconds, firstTokenSeconds: firstTokenSeconds ) } @@ -1762,6 +1823,7 @@ public actor Q35Generator: ChatGenerator { groupSize: Int, bits: Int ) throws { + let checkpointUsesZeroCenteredNorms = try Self.checkpointUsesZeroCenteredRMSNorm(from: resources) let mapper: (String, MLXArray) -> [(String, MLXArray)] = { key, value in guard let mapped = Self.mapTextWeightKey(key) else { return [] } if q35Model.config.tieWordEmbeddings, mapped == "lm_head.weight" { @@ -1775,7 +1837,13 @@ public actor Q35Generator: ChatGenerator { return [(normalizedMapped, Self.normalizedLinearAttentionConv1DWeight(value))] } if Self.isOffsetRMSNormWeight(normalizedMapped) { - return [(normalizedMapped, value - MLXArray(1.0).asType(value.dtype))] + return [( + normalizedMapped, + Self.normalizedRMSNormWeight( + value, + checkpointUsesZeroCenteredNorms: checkpointUsesZeroCenteredNorms + ) + )] } return [(normalizedMapped, value)] } @@ -1792,7 +1860,13 @@ public actor Q35Generator: ChatGenerator { return [(key, Self.normalizedLinearAttentionConv1DWeight(value))] } if Self.isOffsetRMSNormWeight(key) { - return [(key, value - MLXArray(1.0).asType(value.dtype))] + return [( + key, + Self.normalizedRMSNormWeight( + value, + checkpointUsesZeroCenteredNorms: checkpointUsesZeroCenteredNorms + ) + )] } return [(key, value)] } @@ -1842,6 +1916,97 @@ public actor Q35Generator: ChatGenerator { } } + private static func hasMTPWeights(resources: Q35Resources) -> Bool { + let standalone = resources.rootURL.appendingPathComponent("mtp.safetensors") + if FileManager.default.fileExists(atPath: standalone.path) { + return true + } + guard FileManager.default.fileExists(atPath: resources.modelIndexURL.path), + let data = try? Data(contentsOf: resources.modelIndexURL), + let index = try? JSONDecoder().decode(HFSafetensorsIndex.self, from: data) else { + return false + } + return index.weightMap.keys.contains { $0.hasPrefix("mtp.") } + } + + private static func mtpResources(primary resources: Q35Resources) -> Q35Resources? { + if hasMTPWeights(resources: resources) { + return resources + } + let mounted = Q35Resources( + rootURL: resources.rootURL.appendingPathComponent( + Q35Resources.q38MTPComponentPath, + isDirectory: true + ) + ) + return hasMTPWeights(resources: mounted) ? mounted : nil + } + + private func loadMTPWeights( + into mtp: Q35MTPModel, + from resources: Q35Resources, + groupSize: Int, + bits: Int + ) throws { + let standalone = resources.rootURL.appendingPathComponent("mtp.safetensors") + if FileManager.default.fileExists(atPath: standalone.path) { + let metadata = try SafetensorsStreamingLoader.metadata(url: standalone) + if metadata.keys.contains(where: { $0.hasSuffix(".scales") }) { + let arrays = try MLX.loadArrays(url: standalone) + try HFSafetensorsWeightsLoader.applyQuantizedWeightsFromArrays( + arrays, + to: mtp, + groupSize: groupSize, + bits: bits, + keyMapper: { key in + Self.mapMTPWeightKey(key) ?? "__unused__.\(key)" + } + ) + } else { + try SafetensorsStreamingLoader.applyWeightsStreaming( + url: standalone, + to: mtp, + dtype: .bfloat16, + verify: .none, + include: { Self.mapMTPWeightKey($0) != nil }, + mapper: { key, value in + guard let mapped = Self.mapMTPWeightKey(key) else { return [] } + return [(mapped, value)] + }, + batchSize: 32 + ) + } + return + } + + let data = try Data(contentsOf: resources.modelIndexURL) + let index = try JSONDecoder().decode(HFSafetensorsIndex.self, from: data) + let shardFilenames = Self.embeddedMTPShardFilenames(weightMap: index.weightMap) + for filename in shardFilenames { + try HFSafetensorsWeightsLoader.applyWeights( + url: resources.rootURL.appendingPathComponent(filename), + to: mtp, + dtype: .bfloat16, + verify: .none, + mapper: { key, value in + guard let mapped = Self.mapMTPWeightKey(key) else { return [] } + return [(mapped, value)] + } + ) + } + } + + static func embeddedMTPShardFilenames(weightMap: [String: String]) -> [String] { + Array(Set(weightMap.compactMap { key, filename in + key.hasPrefix("mtp.") ? filename : nil + })).sorted() + } + + private static func mapMTPWeightKey(_ key: String) -> String? { + guard key.hasPrefix("mtp.") else { return nil } + return String(key.dropFirst("mtp.".count)) + } + private static func mapTextWeightKey(_ key: String) -> String? { if key.hasPrefix("lm_head.") { return key @@ -1886,6 +2051,58 @@ public actor Q35Generator: ChatGenerator { || key == "model.norm.weight" } + static func normalizedRMSNormWeight( + _ value: MLXArray, + checkpointUsesZeroCenteredNorms: Bool + ) -> MLXArray { + if checkpointUsesZeroCenteredNorms { + return value + } + return value - MLXArray(1.0).asType(value.dtype) + } + + static func checkpointUsesZeroCenteredRMSNorm(from resources: Q35Resources) throws -> Bool { + if FileManager.default.fileExists(atPath: resources.modelIndexURL.path) { + let data = try Data(contentsOf: resources.modelIndexURL) + let index = try JSONDecoder().decode(HFSafetensorsIndex.self, from: data) + let weightKeys = Array(index.weightMap.keys) + if checkpointUsesZeroCenteredRMSNorm(weightKeys: weightKeys, tensorShapes: [:]) { + return true + } + guard let convEntry = index.weightMap.first(where: { key, _ in + key.hasSuffix(".linear_attn.conv1d.weight") + }) else { + return false + } + let shardURL = resources.rootURL.appending(path: convEntry.value) + let metadata = try SafetensorsStreamingLoader.metadata(url: shardURL) + return checkpointUsesZeroCenteredRMSNorm( + weightKeys: weightKeys, + tensorShapes: metadata.mapValues(\.shape) + ) + } + + let metadata = try SafetensorsStreamingLoader.metadata(url: resources.modelWeightsURL) + return checkpointUsesZeroCenteredRMSNorm( + weightKeys: Array(metadata.keys), + tensorShapes: metadata.mapValues(\.shape) + ) + } + + static func checkpointUsesZeroCenteredRMSNorm( + weightKeys: [String], + tensorShapes: [String: [Int]] + ) -> Bool { + if weightKeys.contains(where: { $0.hasPrefix("mtp.") || $0.contains(".mtp.") }) { + return true + } + return tensorShapes.contains { key, shape in + key.hasSuffix(".linear_attn.conv1d.weight") + && shape.count == 3 + && shape.last != 1 + } + } + private static func splitMappedExpertGateUpWeight(_ key: String, _ value: MLXArray) -> [(String, MLXArray)]? { let expertGateUpSuffix = ".mlp.experts.gate_up_proj" guard key.hasSuffix(expertGateUpSuffix), value.ndim == 3 else { @@ -2201,11 +2418,10 @@ public actor Q35Generator: ChatGenerator { spatialMergeSize: Int ) throws -> (tensor: MLXArray, gridTHW: (Int, Int, Int)) { let image = try loadImage(from: imageRef) - let target = Self.qwen3VLTargetSize( + let target = try Self.qwen3VLTargetSize( originalWidth: image.width, originalHeight: image.height, patchSize: patchSize, - temporalPatchSize: visionTower?.temporalPatchSize ?? 2, spatialMergeSize: spatialMergeSize, minPixels: visionMinPixels, maxPixels: visionMaxPixels diff --git a/Sources/MereRunCore/Q35/Q35MTP.swift b/Sources/MereRunCore/Q35/Q35MTP.swift index c51568c8..7646fe02 100644 --- a/Sources/MereRunCore/Q35/Q35MTP.swift +++ b/Sources/MereRunCore/Q35/Q35MTP.swift @@ -81,28 +81,60 @@ final class Q35MTPExperts: Module { } } -final class Q35MTPMoE: Module { - @ModuleInfo(key: "gate") var gate: Linear - @ModuleInfo(key: "experts") var experts: Q35MTPExperts - @ModuleInfo(key: "shared_expert") var sharedExpert: Q35MLP - @ModuleInfo(key: "shared_expert_gate") var sharedExpertGate: Linear +final class Q35MTPFeedForward: Module { + @ModuleInfo(key: "gate") var gate: Linear? + @ModuleInfo(key: "experts") var experts: Q35MTPExperts? + @ModuleInfo(key: "shared_expert") var sharedExpert: Q35MLP? + @ModuleInfo(key: "shared_expert_gate") var sharedExpertGate: Linear? + @ModuleInfo(key: "gate_proj") var gateProj: Linear? + @ModuleInfo(key: "up_proj") var upProj: Linear? + @ModuleInfo(key: "down_proj") var downProj: Linear? private let topK: Int + private let usesMoE: Bool init(config: Q35Config) { let text = config.textConfig + self.usesMoE = text.usesMoE self.topK = max(1, text.numExpertsPerTok) - self._gate.wrappedValue = Linear(text.hiddenSize, text.numExperts, bias: false) - self._experts.wrappedValue = Q35MTPExperts(config: config) - self._sharedExpert.wrappedValue = Q35MLP( - hiddenSize: text.hiddenSize, - intermediateSize: text.sharedExpertIntermediateSize - ) - self._sharedExpertGate.wrappedValue = Linear(text.hiddenSize, 1, bias: false) + + if text.usesMoE { + self._gate.wrappedValue = Linear(text.hiddenSize, text.numExperts, bias: false) + self._experts.wrappedValue = Q35MTPExperts(config: config) + self._sharedExpert.wrappedValue = Q35MLP( + hiddenSize: text.hiddenSize, + intermediateSize: text.sharedExpertIntermediateSize + ) + self._sharedExpertGate.wrappedValue = Linear(text.hiddenSize, 1, bias: false) + self._gateProj.wrappedValue = nil + self._upProj.wrappedValue = nil + self._downProj.wrappedValue = nil + } else { + self._gate.wrappedValue = nil + self._experts.wrappedValue = nil + self._sharedExpert.wrappedValue = nil + self._sharedExpertGate.wrappedValue = nil + self._gateProj.wrappedValue = Linear(text.hiddenSize, text.intermediateSize, bias: false) + self._upProj.wrappedValue = Linear(text.hiddenSize, text.intermediateSize, bias: false) + self._downProj.wrappedValue = Linear(text.intermediateSize, text.hiddenSize, bias: false) + } super.init() } func callAsFunction(_ x: MLXArray) -> MLXArray { + if !usesMoE, + let gateProj, + let upProj, + let downProj { + return downProj(q35MTPSwiglu(gateProj(x), upProj(x))) + } + + guard let gate, + let experts, + let sharedExpert, + let sharedExpertGate else { + return x + } var scores = softmax(gate(x), axis: -1) let k = min(topK, scores.dim(-1)) let indices = argPartition(-scores, kth: k - 1, axis: -1)[.ellipsis, 0.. Bool { - isBonsai27BModelId(modelId) + isQ38ModelId(modelId) + || isBonsai27BModelId(modelId) || modelId == ornith9BModelId || modelId == ornith35BMLXModelId } @@ -64,6 +70,8 @@ public struct Q35Resources: Sendable, Hashable { /// one; callers use it when the user did not set explicit sampling. public static func recommendedSampling(forModelId modelId: String) -> RecommendedSampling? { switch modelId { + case q38TwentySevenBModelId, q38TwentySevenB4BitModelId: + return RecommendedSampling(temperature: 1.0, topP: 0.95, topK: 20) case bonsai27B1BitModelId, bonsai27B2BitModelId: return RecommendedSampling(temperature: 0.7, topP: 0.95, topK: 20) case ornith9BModelId, ornith35BMLXModelId: @@ -75,6 +83,21 @@ public struct Q35Resources: Sendable, Hashable { public static let q36NanoUpstreamRepoId = "mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit" public static let q36NanoUpstreamRevision = "63d520640ca7461f31ba66104612135770090340" + public static let q38TwentySevenBUpstreamRepoId = "Qwen/Qwen3.8-27B" + public static let q38TwentySevenBUpstreamRevision = "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0" + public static let q38TwentySevenBEstimatedDownloadBytes: Int64 = 55_586_114_863 + public static let q38TwentySevenB4BitUpstreamRepoId = "lmstudio-community/Qwen3.8-27B-MLX-4bit" + public static let q38TwentySevenB4BitUpstreamRevision = "6067b15cf581666a4aecf6af3afaba4bb5efc20c" + public static let q38TwentySevenB4BitEstimatedDownloadBytes: Int64 = 19_473_823_081 + public static let q38MTPComponentPath = "mtp" + public static let q38MTPComponentSnapshotPatterns = [ + "LICENSE", + "model.safetensors.index.json", + "model-00018-of-00018.safetensors", + ] + public static let q38TwentySevenBContextLength = 262_144 + public static let q38TwentySevenBVisionMinPixels = 65_536 + public static let q38TwentySevenBVisionMaxPixels = 16_777_216 public static let bonsai27B1BitUpstreamRepoId = "prism-ml/Bonsai-27B-mlx-1bit" public static let bonsai27B1BitUpstreamRevision = "ef22f239c670078e1507f9769bcaa66657332b96" public static let bonsai27B1BitEstimatedDownloadBytes: Int64 = 5_128_837_600 @@ -100,6 +123,29 @@ public struct Q35Resources: Sendable, Hashable { upstreamRepoId: q36NanoUpstreamRepoId, upstreamRevision: q36NanoUpstreamRevision ), + q38TwentySevenBModelId: Profile( + modelId: q38TwentySevenBModelId, + upstreamRepoId: q38TwentySevenBUpstreamRepoId, + upstreamRevision: q38TwentySevenBUpstreamRevision, + snapshotPatterns: snapshotPatterns + [ + "generation_config.json", + "preprocessor_config.json", + "video_preprocessor_config.json", + "merges.txt", + "vocab.json", + ] + ), + q38TwentySevenB4BitModelId: Profile( + modelId: q38TwentySevenB4BitModelId, + upstreamRepoId: q38TwentySevenB4BitUpstreamRepoId, + upstreamRevision: q38TwentySevenB4BitUpstreamRevision, + snapshotPatterns: snapshotPatterns + [ + "generation_config.json", + "preprocessor_config.json", + "video_preprocessor_config.json", + "vocab.json", + ] + ), bonsai27B1BitModelId: Profile( modelId: bonsai27B1BitModelId, upstreamRepoId: bonsai27B1BitUpstreamRepoId, @@ -145,6 +191,8 @@ public struct Q35Resources: Sendable, Hashable { public static func defaultContextLength(forModelId modelId: String) -> Int { switch modelId { + case q38TwentySevenBModelId, q38TwentySevenB4BitModelId: + q38TwentySevenBContextLength case bonsai27B1BitModelId: bonsai27B1BitContextLength case bonsai27B2BitModelId: @@ -154,6 +202,17 @@ public struct Q35Resources: Sendable, Hashable { } } + public static func visionPixelBounds(forModelId modelId: String) -> (minimum: Int, maximum: Int) { + if isQ38ModelId(modelId) { + return (q38TwentySevenBVisionMinPixels, q38TwentySevenBVisionMaxPixels) + } + return (Q35Generator.qwen3VLMinPixels, Q35Generator.qwen3VLMaxPixels) + } + + public static func isQ38ModelId(_ modelId: String) -> Bool { + modelId == q38TwentySevenBModelId || modelId == q38TwentySevenB4BitModelId + } + public static let snapshotPatterns = [ "LICENSE*", "NOTICE*", @@ -180,6 +239,7 @@ public struct Q35Resources: Sendable, Hashable { public var tokenizerURL: URL { rootURL.appending(path: "tokenizer.json") } public var tokenizerConfigURL: URL { rootURL.appending(path: "tokenizer_config.json") } public var chatTemplateURL: URL { rootURL.appending(path: "chat_template.jinja") } + public var generationConfigURL: URL { rootURL.appending(path: "generation_config.json") } public var processorConfigURL: URL { rootURL.appending(path: "processor_config.json") } public func validate(fileManager: FileManager = .default) -> [URL] { @@ -205,6 +265,16 @@ public struct Q35Resources: Sendable, Hashable { return missing } + public func validateQ38MTPComponent(fileManager: FileManager = .default) -> [URL] { + let componentRoot = rootURL.appendingPathComponent( + Self.q38MTPComponentPath, + isDirectory: true + ) + return Self.q38MTPComponentSnapshotPatterns + .map { componentRoot.appendingPathComponent($0, isDirectory: false) } + .filter { !fileManager.fileExists(atPath: $0.path) } + } + public static func normalizedRootURL(_ rootURL: URL, fileManager: FileManager = .default) -> URL { let standardized = rootURL.standardizedFileURL let directConfig = standardized.appendingPathComponent("config.json") diff --git a/Sources/MereRunCore/Q35/Q35TokenizerAndTemplate.swift b/Sources/MereRunCore/Q35/Q35TokenizerAndTemplate.swift index d313d36d..a746c8bb 100644 --- a/Sources/MereRunCore/Q35/Q35TokenizerAndTemplate.swift +++ b/Sources/MereRunCore/Q35/Q35TokenizerAndTemplate.swift @@ -1,6 +1,20 @@ import Foundation @preconcurrency import Tokenizers +enum Q35TokenizerAndTemplateError: LocalizedError { + case missingImageToken + case imageTokenCountMismatch(expected: Int, actual: Int) + + var errorDescription: String? { + switch self { + case .missingImageToken: + "Qwen-family tokenizer is missing the image placeholder token." + case .imageTokenCountMismatch(let expected, let actual): + "Qwen-family prompt rendered \(actual) image placeholders for \(expected) encoded images." + } + } +} + public struct Q35TokenizerAndTemplate { public let tokenizer: QwenTokenizer @@ -21,31 +35,30 @@ public struct Q35TokenizerAndTemplate { maxLength: Int, imageTokenCounts: [Int] = [] ) throws -> [Int] { - if !imageTokenCounts.isEmpty { - var encoded = tokenizer.encodeText( - Self.renderPrompt( - messages: messages, - tools: tools, - addGenerationPrompt: addGenerationPrompt, - includeThinking: includeThinking, - imageTokenCounts: imageTokenCounts - ) - ) - let targetLength = min(maxLength, tokenizer.maxLength) - if encoded.count > targetLength { - encoded = Array(encoded.suffix(targetLength)) - } - return encoded - } - let toolSpecs: [ToolSpec]? = tools?.isEmpty == false ? tools!.map { $0.toToolSpec() } : nil - return try tokenizer.encodeChatTemplate( + var encoded = try tokenizer.encodeChatTemplate( messages: Self.renderMessages(messages), tools: toolSpecs, addGenerationPrompt: addGenerationPrompt, includeThinking: includeThinking, - maxLength: maxLength + maxLength: tokenizer.maxLength ) + if !imageTokenCounts.isEmpty { + guard let imageTokenId = tokenizer.imageTokenId else { + throw Q35TokenizerAndTemplateError.missingImageToken + } + encoded = try Self.expandingImageTokenIds( + encoded, + imageTokenId: imageTokenId, + imageTokenCounts: imageTokenCounts + ) + } + + let targetLength = min(maxLength, tokenizer.maxLength) + if encoded.count > targetLength { + encoded = Array(encoded.suffix(targetLength)) + } + return encoded } public func decode(tokens: [Int]) -> String { @@ -93,9 +106,79 @@ public struct Q35TokenizerAndTemplate { rendered["content"] = message.content } + if let reasoning = message.reasoningContent, !reasoning.isEmpty { + rendered["reasoning_content"] = reasoning + } + if let name = message.name, !name.isEmpty { + rendered["name"] = name + } + if let toolCallID = message.toolCallID, !toolCallID.isEmpty { + rendered["tool_call_id"] = toolCallID + } + if let calls = message.toolCalls, !calls.isEmpty { + rendered["tool_calls"] = calls.map { call -> [String: any Sendable] in + var result: [String: any Sendable] = [ + "function": [ + "name": call.name, + "arguments": call.arguments.mapValues(renderJSONValue), + ] as [String: any Sendable], + ] + if let id = call.id, !id.isEmpty { + result["id"] = id + } + return result + } + } + return rendered } + static func expandingImageTokenIds( + _ tokenIds: [Int], + imageTokenId: Int, + imageTokenCounts: [Int] + ) throws -> [Int] { + var expanded: [Int] = [] + expanded.reserveCapacity(tokenIds.count + imageTokenCounts.reduce(0, +)) + var imageIndex = 0 + + for tokenId in tokenIds { + guard tokenId == imageTokenId else { + expanded.append(tokenId) + continue + } + guard imageIndex < imageTokenCounts.count else { + throw Q35TokenizerAndTemplateError.imageTokenCountMismatch( + expected: imageTokenCounts.count, + actual: imageIndex + 1 + ) + } + expanded.append( + contentsOf: repeatElement(imageTokenId, count: max(1, imageTokenCounts[imageIndex])) + ) + imageIndex += 1 + } + + guard imageIndex == imageTokenCounts.count else { + throw Q35TokenizerAndTemplateError.imageTokenCountMismatch( + expected: imageTokenCounts.count, + actual: imageIndex + ) + } + return expanded + } + + private static func renderJSONValue(_ value: OpenAIJSONValue) -> any Sendable { + switch value { + case .string(let value): value + case .number(let value): value + case .bool(let value): value + case .object(let value): value.mapValues(renderJSONValue) + case .array(let value): value.map(renderJSONValue) + case .null: Optional.none + } + } + static func renderPrompt( messages: [ChatMessage], tools: [ToolDefinition]? = nil, diff --git a/Sources/MereRunCore/Q35/README.md b/Sources/MereRunCore/Q35/README.md index b5933ffc..71b45d02 100644 --- a/Sources/MereRunCore/Q35/README.md +++ b/Sources/MereRunCore/Q35/README.md @@ -1,11 +1,25 @@ # Q35 -Qwen 3.5/3.6 hybrid MoE text and vision-language runtime. +Qwen 3.5/3.6/3.8 dense and hybrid MoE text and vision-language runtime. - `Q35Config.swift`: typed text/vision configuration. -- `Q35TokenizerAndTemplate.swift`: prompt rendering and tokenization. +- `Q35TokenizerAndTemplate.swift`: checkpoint-native chat-template rendering, + image-token expansion, and tokenization. - `Q35Model.swift`: native model entry point. - Attention and MoE files own model math only. Keep tokenizer/tool template compatibility isolated here; model layers should not know about CLI or managed-model concerns. + +Official Hugging Face Qwen 3.5/3.8 checkpoints store zero-centered RMSNorm +offsets, while converted MLX checkpoints store direct scales. The loader detects +the checkpoint layout from embedded MTP keys or PyTorch Conv1d shapes and keeps +both conventions compatible with the native offset RMSNorm module. + +The official Qwen3.8 27B shards embed a dense one-layer MTP head. The loader +reads only the shards that contain `mtp.*` tensors, maps its dense SwiGLU layout, +and also discovers the same pinned official shard under `mtp/` beside the +managed 4-bit target. `MERERUN_Q35_MTP_SPECULATION=1` enables greedy speculation +from short prompts. It stays opt-in because multi-token target verification can +choose a different greedy path from serial target decode. Hybrid MoE Qwen models +keep the existing adaptive long-context threshold. diff --git a/Tests/MereRunCLITests/ModelBenchmarkCommandTests.swift b/Tests/MereRunCLITests/ModelBenchmarkCommandTests.swift index cad7ba73..497f09ae 100644 --- a/Tests/MereRunCLITests/ModelBenchmarkCommandTests.swift +++ b/Tests/MereRunCLITests/ModelBenchmarkCommandTests.swift @@ -191,6 +191,35 @@ final class ModelBenchmarkCommandTests: XCTestCase { ]) } + func testCodeBenchmarkAcceptsQ38CodeGenerationLane() throws { + for modelID in [ + Q35Resources.q38TwentySevenBModelId, + Q35Resources.q38TwentySevenB4BitModelId, + ] { + let spec = try XCTUnwrap(ManagedModelCatalog.spec(for: modelID)) + XCTAssertEqual(spec.category, .visionChat) + XCTAssertTrue(ModelBenchmarkCode.supportsCodingBenchmark(spec)) + } + } + + func testCodeBenchmarkScoresOnlyVisibleCodeAfterImplicitThinkingPrefix() { + let response = ChatResponse( + response: """ + from typing? The prompt already imports it. Return only code. + + + def has_close_elements(numbers, threshold): + return False + """, + tokensGenerated: 32 + ) + + let scored = ModelBenchmarkCode.scoredCodeResponse(response) + + XCTAssertFalse(scored.contains("from typing?")) + XCTAssertTrue(scored.hasPrefix("def has_close_elements")) + } + func testCodeBenchmarkParsesOverrides() throws { let cmd = try ModelBenchmarkCode.parse([ "--models", "text-agent-ornith-35b-mlx,text-code-north-mini", diff --git a/Tests/MereRunCLITests/TextChatCommandParsingTests.swift b/Tests/MereRunCLITests/TextChatCommandParsingTests.swift index decaa213..36a17ead 100644 --- a/Tests/MereRunCLITests/TextChatCommandParsingTests.swift +++ b/Tests/MereRunCLITests/TextChatCommandParsingTests.swift @@ -383,9 +383,11 @@ final class TextChatCommandParsingTests: XCTestCase { XCTAssertEqual(explicit.minP, 0.05) } - func testOrnithLanesDefaultToThinkingAndRecommendedSampling() { + func testQwenAgentAndQ38LanesDefaultToThinkingAndRecommendedSampling() { XCTAssertTrue(Q35Resources.thinkingDefault(forModelId: Q35Resources.ornith35BMLXModelId)) XCTAssertTrue(Q35Resources.thinkingDefault(forModelId: Q35Resources.ornith9BModelId)) + XCTAssertTrue(Q35Resources.thinkingDefault(forModelId: Q35Resources.q38TwentySevenBModelId)) + XCTAssertTrue(Q35Resources.thinkingDefault(forModelId: Q35Resources.q38TwentySevenB4BitModelId)) XCTAssertFalse(Q35Resources.thinkingDefault(forModelId: Q35Resources.q36NanoModelId)) XCTAssertFalse(Q35Resources.thinkingDefault(forModelId: Gemma4Resources.twelveB4BitModelId)) @@ -393,6 +395,14 @@ final class TextChatCommandParsingTests: XCTestCase { XCTAssertEqual(sampling?.temperature, 1.0) XCTAssertEqual(sampling?.topP, 0.95) XCTAssertEqual(sampling?.topK, 20) + let q38Sampling = Q35Resources.recommendedSampling(forModelId: Q35Resources.q38TwentySevenBModelId) + XCTAssertEqual(q38Sampling?.temperature, 1.0) + XCTAssertEqual(q38Sampling?.topP, 0.95) + XCTAssertEqual(q38Sampling?.topK, 20) + XCTAssertEqual( + Q35Resources.recommendedSampling(forModelId: Q35Resources.q38TwentySevenB4BitModelId), + q38Sampling + ) XCTAssertNil(Q35Resources.recommendedSampling(forModelId: Q35Resources.q36NanoModelId)) } diff --git a/Tests/MereRunCoreTests/ManagedModelCatalogTests.swift b/Tests/MereRunCoreTests/ManagedModelCatalogTests.swift index 927d5864..efcebd3f 100644 --- a/Tests/MereRunCoreTests/ManagedModelCatalogTests.swift +++ b/Tests/MereRunCoreTests/ManagedModelCatalogTests.swift @@ -764,6 +764,80 @@ final class ManagedModelCatalogTests: XCTestCase { XCTAssertEqual(spec.hubFallback?.patterns.contains("*.safetensors"), true) } + func testQ38TwentySevenBUsesPinnedOfficialBF16Source() throws { + let spec = try XCTUnwrap(ManagedModelCatalog.spec(for: Q35Resources.q38TwentySevenBModelId)) + + XCTAssertEqual(ModelResolver.ModelID.q38TwentySevenB.rawValue, spec.id) + XCTAssertEqual(spec.category, .visionChat) + XCTAssertEqual(spec.installShape, .directoryRoot) + XCTAssertEqual(spec.hubFallback?.repoId, Q35Resources.q38TwentySevenBUpstreamRepoId) + XCTAssertEqual(spec.hubFallback?.revision, Q35Resources.q38TwentySevenBUpstreamRevision) + XCTAssertEqual(spec.upstreamRepoId, Q35Resources.q38TwentySevenBUpstreamRepoId) + XCTAssertEqual(spec.upstreamRevision, Q35Resources.q38TwentySevenBUpstreamRevision) + XCTAssertEqual(spec.validationKind, .q35) + XCTAssertEqual(spec.defaultRuntimeServingEngine, .textChatQ36) + XCTAssertEqual(spec.estimatedDownloadBytes, 55_586_114_863) + XCTAssertFalse(spec.runtimeAutoDownloadAllowed) + XCTAssertTrue(spec.defaultCLICommands.contains("model benchmark code")) + XCTAssertEqual(spec.hubFallback?.patterns.contains("generation_config.json"), true) + XCTAssertEqual(spec.hubFallback?.patterns.contains("preprocessor_config.json"), true) + XCTAssertEqual(spec.hubFallback?.patterns.contains("video_preprocessor_config.json"), true) + } + + func testQ38TwentySevenB4BitUsesPinnedConversionAndOfficialMTPShard() throws { + let spec = try XCTUnwrap( + ManagedModelCatalog.spec(for: Q35Resources.q38TwentySevenB4BitModelId) + ) + + XCTAssertEqual(ModelResolver.ModelID.q38TwentySevenB4Bit.rawValue, spec.id) + XCTAssertEqual(spec.category, .visionChat) + XCTAssertEqual(spec.installShape, .directoryRoot) + XCTAssertEqual(spec.hubFallback?.repoId, Q35Resources.q38TwentySevenB4BitUpstreamRepoId) + XCTAssertEqual(spec.hubFallback?.revision, Q35Resources.q38TwentySevenB4BitUpstreamRevision) + XCTAssertEqual(spec.mountedHubFallbacks.count, 1) + XCTAssertEqual(spec.mountedHubFallbacks.first?.destinationPath, Q35Resources.q38MTPComponentPath) + XCTAssertEqual( + spec.mountedHubFallbacks.first?.hubFallback.repoId, + Q35Resources.q38TwentySevenBUpstreamRepoId + ) + XCTAssertEqual( + spec.mountedHubFallbacks.first?.hubFallback.revision, + Q35Resources.q38TwentySevenBUpstreamRevision + ) + XCTAssertEqual( + spec.mountedHubFallbacks.first?.hubFallback.patterns, + Q35Resources.q38MTPComponentSnapshotPatterns + ) + XCTAssertEqual(spec.validationKind, .q35) + XCTAssertEqual(spec.defaultRuntimeServingEngine, .textChatQ36) + XCTAssertEqual(spec.estimatedDownloadBytes, 19_473_823_081) + XCTAssertFalse(spec.runtimeAutoDownloadAllowed) + XCTAssertTrue(spec.defaultCLICommands.contains("model benchmark code")) + } + + func testQ38TwentySevenB4BitValidationRequiresMountedMTPFiles() throws { + let spec = try XCTUnwrap( + ManagedModelCatalog.spec(for: Q35Resources.q38TwentySevenB4BitModelId) + ) + let root = try makeTemporaryDirectory() + defer { try? FileManager.default.removeItem(at: root) } + for filename in ["config.json", "model.safetensors", "tokenizer.json", "tokenizer_config.json"] { + try Data().write(to: root.appendingPathComponent(filename)) + } + + XCTAssertEqual( + Set(spec.missingPaths(in: root).map(\.lastPathComponent)), + Set(Q35Resources.q38MTPComponentSnapshotPatterns) + ) + + let mtpRoot = root.appendingPathComponent(Q35Resources.q38MTPComponentPath, isDirectory: true) + try FileManager.default.createDirectory(at: mtpRoot, withIntermediateDirectories: true) + for filename in Q35Resources.q38MTPComponentSnapshotPatterns { + try Data().write(to: mtpRoot.appendingPathComponent(filename)) + } + XCTAssertTrue(spec.missingPaths(in: root).isEmpty) + } + func testBonsai27BUsesPinnedPackedOneBitQ35Source() throws { let spec = try XCTUnwrap(ManagedModelCatalog.spec(for: Q35Resources.bonsai27B1BitModelId)) diff --git a/Tests/MereRunCoreTests/ManagedModelSupportTests.swift b/Tests/MereRunCoreTests/ManagedModelSupportTests.swift index 0d31ed8a..30be524e 100644 --- a/Tests/MereRunCoreTests/ManagedModelSupportTests.swift +++ b/Tests/MereRunCoreTests/ManagedModelSupportTests.swift @@ -202,6 +202,56 @@ final class ManagedModelSupportTests: XCTestCase { XCTAssertEqual(report.reasons, []) } + func testQ38TwentySevenBRequiresHighMemoryAppleSilicon() throws { + let spec = try XCTUnwrap(ManagedModelCatalog.spec(for: Q35Resources.q38TwentySevenBModelId)) + let undersized = MereRunMachineProfile( + physicalMemoryBytes: 48 * 1_073_741_824, + processorName: "M4 Max", + isAppleSiliconMac: true + ) + let recommended = MereRunMachineProfile( + physicalMemoryBytes: 96 * 1_073_741_824, + processorName: "M4 Max", + isAppleSiliconMac: true + ) + + let rejected = ManagedModelCapabilityCatalog.support(for: spec, on: undersized) + let accepted = ManagedModelCapabilityCatalog.support(for: spec, on: recommended) + + XCTAssertFalse(rejected.isSupported) + XCTAssertTrue(rejected.reasons.joined(separator: " ").contains("Requires at least 64 GB")) + XCTAssertTrue(accepted.isSupported) + XCTAssertTrue(accepted.meetsRecommendedMemory) + XCTAssertEqual(accepted.descriptor.minimumUnifiedMemoryGB, 64) + XCTAssertEqual(accepted.descriptor.recommendedUnifiedMemoryGB, 96) + } + + func testQ38TwentySevenB4BitSupportsThirtyTwoGBAndRecommendsFortyEightGB() throws { + let spec = try XCTUnwrap( + ManagedModelCatalog.spec(for: Q35Resources.q38TwentySevenB4BitModelId) + ) + let minimum = MereRunMachineProfile( + physicalMemoryBytes: 32 * 1_073_741_824, + processorName: "M4 Pro", + isAppleSiliconMac: true + ) + let recommended = MereRunMachineProfile( + physicalMemoryBytes: 48 * 1_073_741_824, + processorName: "M4 Max", + isAppleSiliconMac: true + ) + + let minimumReport = ManagedModelCapabilityCatalog.support(for: spec, on: minimum) + let recommendedReport = ManagedModelCapabilityCatalog.support(for: spec, on: recommended) + + XCTAssertTrue(minimumReport.isSupported) + XCTAssertFalse(minimumReport.meetsRecommendedMemory) + XCTAssertTrue(recommendedReport.isSupported) + XCTAssertTrue(recommendedReport.meetsRecommendedMemory) + XCTAssertEqual(recommendedReport.descriptor.minimumUnifiedMemoryGB, 32) + XCTAssertEqual(recommendedReport.descriptor.recommendedUnifiedMemoryGB, 48) + } + func testBonsai27BIsSupportedOnSixteenGB() throws { let spec = try XCTUnwrap(ManagedModelCatalog.spec(for: Q35Resources.bonsai27B1BitModelId)) let machine = MereRunMachineProfile( diff --git a/Tests/MereRunCoreTests/MereRunModelManifestTests.swift b/Tests/MereRunCoreTests/MereRunModelManifestTests.swift index 250af5d0..12675f71 100644 --- a/Tests/MereRunCoreTests/MereRunModelManifestTests.swift +++ b/Tests/MereRunCoreTests/MereRunModelManifestTests.swift @@ -215,6 +215,51 @@ final class MereRunModelManifestTests: MereRunCoreTestCase { ) } + func testQ38TwentySevenBTemplatePinsOfficialBF16Checkpoint() throws { + let manifest = MereRunModelManifest.template( + for: .q38TwentySevenB, + createdAt: Date(timeIntervalSince1970: 0) + ) + + XCTAssertEqual(manifest.id, Q35Resources.q38TwentySevenBModelId) + XCTAssertEqual(manifest.engine, .qwen35HybridMoE) + XCTAssertEqual(manifest.family, .qwen) + XCTAssertEqual(manifest.precision, .bf16) + XCTAssertNil(manifest.quantization) + XCTAssertEqual(Set(manifest.supports ?? []), Set([.chat, .codeGeneration, .visionChat])) + XCTAssertEqual( + manifest.upstreamRepoId, + "\(Q35Resources.q38TwentySevenBUpstreamRepoId)@\(Q35Resources.q38TwentySevenBUpstreamRevision)" + ) + } + + func testQ38TwentySevenB4BitTemplateRecordsTargetAndMTPSources() throws { + let manifest = MereRunModelManifest.template( + for: .q38TwentySevenB4Bit, + createdAt: Date(timeIntervalSince1970: 0) + ) + + XCTAssertEqual(manifest.id, Q35Resources.q38TwentySevenB4BitModelId) + XCTAssertEqual(manifest.engine, .qwen35HybridMoE) + XCTAssertEqual(manifest.family, .qwen) + XCTAssertEqual(manifest.precision, .int4) + XCTAssertEqual(manifest.quantization?.bits, 4) + XCTAssertEqual(manifest.quantization?.groupSize, 64) + XCTAssertEqual(manifest.quantization?.scheme, "mlx-affine") + XCTAssertEqual(Set(manifest.supports ?? []), Set([.chat, .codeGeneration, .visionChat])) + XCTAssertEqual( + manifest.upstreamRepoId, + "\(Q35Resources.q38TwentySevenB4BitUpstreamRepoId)" + + "@\(Q35Resources.q38TwentySevenB4BitUpstreamRevision)" + ) + XCTAssertEqual(manifest.sources?.count, 2) + XCTAssertEqual(manifest.sources?.first?.role, "primary") + XCTAssertEqual(manifest.sources?.first?.repository, Q35Resources.q38TwentySevenB4BitUpstreamRepoId) + XCTAssertEqual(manifest.sources?.last?.role, "component") + XCTAssertEqual(manifest.sources?.last?.repository, Q35Resources.q38TwentySevenBUpstreamRepoId) + XCTAssertEqual(manifest.sources?.last?.destinationPath, Q35Resources.q38MTPComponentPath) + } + func testFalconPerceptionTemplateHasExpectedMetadata() throws { let manifest = MereRunModelManifest.template(for: .visionGroundFalconPerception, createdAt: Date(timeIntervalSince1970: 0)) diff --git a/Tests/MereRunCoreTests/Q35ConfigDecodingTests.swift b/Tests/MereRunCoreTests/Q35ConfigDecodingTests.swift index a48d050b..3b9ad10f 100644 --- a/Tests/MereRunCoreTests/Q35ConfigDecodingTests.swift +++ b/Tests/MereRunCoreTests/Q35ConfigDecodingTests.swift @@ -85,12 +85,11 @@ final class Q35ConfigDecodingTests: MereRunCoreTestCase { XCTAssertTrue(rendered.contains("<|vision_start|><|image_pad|><|image_pad|><|image_pad|><|vision_end|>")) } - func testQ35Qwen3VLTargetSizeUsesInfinityParserPixelBudget() { - let target = Q35Generator.qwen3VLTargetSize( + func testQ35Qwen3VLTargetSizeUsesUpstreamSpatialPixelBudget() throws { + let target = try Q35Generator.qwen3VLTargetSize( originalWidth: 2_108, originalHeight: 1_094, patchSize: 16, - temporalPatchSize: 2, spatialMergeSize: 2 ) @@ -99,6 +98,59 @@ final class Q35ConfigDecodingTests: MereRunCoreTestCase { XCTAssertEqual((target.width / 16) * (target.height / 16) / 4, 2_244) } + func testQ38ImageResizeDoesNotCountDuplicatedTemporalPatch() throws { + let target = try Q35Generator.qwen3VLTargetSize( + originalWidth: 3_072, + originalHeight: 3_072, + patchSize: 16, + spatialMergeSize: 2, + minPixels: Q35Resources.q38TwentySevenBVisionMinPixels, + maxPixels: Q35Resources.q38TwentySevenBVisionMaxPixels + ) + + XCTAssertEqual(target.width, 3_072) + XCTAssertEqual(target.height, 3_072) + } + + func testQ38ImageResizeUpscalesFromPublishedSpatialMinimum() throws { + let target = try Q35Generator.qwen3VLTargetSize( + originalWidth: 256, + originalHeight: 128, + patchSize: 16, + spatialMergeSize: 2, + minPixels: Q35Resources.q38TwentySevenBVisionMinPixels, + maxPixels: Q35Resources.q38TwentySevenBVisionMaxPixels + ) + + XCTAssertEqual(target.width, 384) + XCTAssertEqual(target.height, 192) + } + + func testQ35ImageResizeUsesPythonTieToEvenRounding() throws { + let target = try Q35Generator.qwen3VLTargetSize( + originalWidth: 80, + originalHeight: 80, + patchSize: 16, + spatialMergeSize: 2, + minPixels: 1, + maxPixels: 1_000_000 + ) + + XCTAssertEqual(target.width, 64) + XCTAssertEqual(target.height, 64) + } + + func testQ35ImageResizeRejectsUpstreamUnsupportedAspectRatio() { + XCTAssertThrowsError( + try Q35Generator.qwen3VLTargetSize( + originalWidth: 6_432, + originalHeight: 32, + patchSize: 16, + spatialMergeSize: 2 + ) + ) + } + func testQ35LinearAttentionConvWeightsKeepMLXLayout() { let mlxLayout = MLXArray.zeros([8, 4, 1]) let normalizedMLX = Q35Generator.normalizedLinearAttentionConv1DWeight(mlxLayout) @@ -205,6 +257,122 @@ final class Q35ConfigDecodingTests: MereRunCoreTestCase { XCTAssertFalse(Q35Generator.isOffsetRMSNormWeight("model.layers.0.linear_attn.norm.weight")) } + func testQ35OfficialCheckpointKeepsZeroCenteredRMSNormWeights() { + let published = MLXArray([Float(-0.125), 0.25]) + let normalized = Q35Generator.normalizedRMSNormWeight( + published, + checkpointUsesZeroCenteredNorms: true + ) + MLX.eval(normalized) + + XCTAssertEqual(normalized[0].item(Float.self), -0.125, accuracy: 1e-6) + XCTAssertEqual(normalized[1].item(Float.self), 0.25, accuracy: 1e-6) + } + + func testQ35ConvertedCheckpointConvertsDirectRMSNormScalesToOffsets() { + let converted = MLXArray([Float(0.875), 1.25]) + let normalized = Q35Generator.normalizedRMSNormWeight( + converted, + checkpointUsesZeroCenteredNorms: false + ) + MLX.eval(normalized) + + XCTAssertEqual(normalized[0].item(Float.self), -0.125, accuracy: 1e-6) + XCTAssertEqual(normalized[1].item(Float.self), 0.25, accuracy: 1e-6) + } + + func testQ35OfficialCheckpointLayoutDetectedFromEmbeddedMTP() { + XCTAssertTrue(Q35Generator.checkpointUsesZeroCenteredRMSNorm( + weightKeys: [ + "model.language_model.layers.0.input_layernorm.weight", + "model.mtp.pre_fc_norm_hidden.weight", + ], + tensorShapes: [:] + )) + } + + func testQ35OfficialCheckpointLayoutDetectedFromPyTorchConv1D() { + let key = "model.language_model.layers.0.linear_attn.conv1d.weight" + XCTAssertTrue(Q35Generator.checkpointUsesZeroCenteredRMSNorm( + weightKeys: [key], + tensorShapes: [key: [12_288, 1, 4]] + )) + XCTAssertFalse(Q35Generator.checkpointUsesZeroCenteredRMSNorm( + weightKeys: [key], + tensorShapes: [key: [12_288, 4, 1]] + )) + } + + func testQ35ImageTokenExpansionPreservesCanonicalTemplateTokens() throws { + let expanded = try Q35TokenizerAndTemplate.expandingImageTokenIds( + [10, 248_056, 20, 248_056, 30], + imageTokenId: 248_056, + imageTokenCounts: [3, 2] + ) + + XCTAssertEqual(expanded, [10, 248_056, 248_056, 248_056, 20, 248_056, 248_056, 30]) + } + + func testQ35ImageTokenExpansionRejectsTemplateImageMismatch() { + XCTAssertThrowsError( + try Q35TokenizerAndTemplate.expandingImageTokenIds( + [10, 20], + imageTokenId: 248_056, + imageTokenCounts: [3] + ) + ) + } + + func testQ35TemplateMessagesPreserveReasoningAndToolCalls() throws { + let rendered = Q35TokenizerAndTemplate.renderMessages([ + ChatMessage( + role: .assistant, + content: "I will inspect it.", + reasoningContent: "Need the exact file.", + toolCalls: [ + ChatMessageToolCall( + id: "call_1", + name: "inspect_file", + arguments: ["path": .string("/tmp/input.png")] + ), + ] + ), + ]) + let message = try XCTUnwrap(rendered.first) + XCTAssertEqual(message["reasoning_content"] as? String, "Need the exact file.") + let calls = try XCTUnwrap(message["tool_calls"] as? [[String: any Sendable]]) + let function = try XCTUnwrap(calls.first?["function"] as? [String: any Sendable]) + XCTAssertEqual(function["name"] as? String, "inspect_file") + } + + func testQ38PinnedTokenizerRendersCanonicalVisionPromptWhenAvailable() throws { + guard let rootPath = ProcessInfo.processInfo.environment["MERERUN_Q38_TOKENIZER_ROOT"] else { + throw XCTSkip("Set MERERUN_Q38_TOKENIZER_ROOT to run pinned Qwen3.8 tokenizer parity.") + } + let template = try Q35TokenizerAndTemplate.load( + from: URL(fileURLWithPath: rootPath), + maxLengthOverride: Q35Resources.q38TwentySevenBContextLength + ) + let tokens = try template.encodeForGeneration( + messages: [ChatMessage( + role: .user, + content: "Read it.", + imageUrl: "/tmp/page.png" + )], + addGenerationPrompt: true, + includeThinking: true, + maxLength: Q35Resources.q38TwentySevenBContextLength, + imageTokenCounts: [3] + ) + let decoded = template.decode(tokens: tokens) + + XCTAssertTrue(decoded.contains("Reasoning effort is set to xhigh.")) + XCTAssertTrue(decoded.contains( + "<|vision_start|><|image_pad|><|image_pad|><|image_pad|><|vision_end|>Read it." + )) + XCTAssertTrue(decoded.hasSuffix("<|im_start|>assistant\n\n")) + } + func testQ35TemplateLeavesThinkingOpenWhenRequested() { let rendered = Q35TokenizerAndTemplate.renderPrompt( messages: [ChatMessage(role: .user, content: "Explain.")], @@ -339,9 +507,16 @@ final class Q35ConfigDecodingTests: MereRunCoreTestCase { XCTAssertEqual(probs.sum().item(Float.self), 1, accuracy: 0.0001) } - func testQ35MTPDraftLogitsSupportsDenseExpertWeightLayout() throws { + func testQ35MTPDraftLogitsSupportsDenseFeedForwardWeightLayout() throws { MLXRandom.seed(39) - let config = try decodeConfig(makeTinyRuntimeConfig(layerTypes: ["full_attention"])) + var configObject = makeTinyRuntimeConfig(layerTypes: ["full_attention"]) + var textConfig = configObject["text_config"] as? [String: Any] ?? [:] + textConfig.removeValue(forKey: "num_experts") + textConfig.removeValue(forKey: "num_experts_per_tok") + textConfig.removeValue(forKey: "moe_intermediate_size") + textConfig.removeValue(forKey: "shared_expert_intermediate_size") + configObject["text_config"] = textConfig + let config = try decodeConfig(configObject) let model = Q35Model(config: config) let mtp = Q35MTPModel(config: config) let tokens = [1, 2, 3] @@ -364,6 +539,17 @@ final class Q35ConfigDecodingTests: MereRunCoreTestCase { XCTAssertTrue(MLX.max(MLX.abs(draftLogits.asType(.float32))).item(Float.self).isFinite) } + func testQ35EmbeddedMTPWeightsSelectOnlyContainingShards() { + let shards = Q35Generator.embeddedMTPShardFilenames(weightMap: [ + "model.language_model.layers.0.mlp.down_proj.weight": "model-00001.safetensors", + "mtp.layers.0.mlp.down_proj.weight": "model-00018.safetensors", + "mtp.norm.weight": "model-00018.safetensors", + "mtp.fc.weight": "model-00017.safetensors", + ]) + + XCTAssertEqual(shards, ["model-00017.safetensors", "model-00018.safetensors"]) + } + func testQ35MTPDraftBlockReturnsRequestedGreedyTokens() throws { MLXRandom.seed(40) let config = try decodeConfig(makeTinyRuntimeConfig(layerTypes: ["full_attention"])) @@ -416,6 +602,27 @@ final class Q35ConfigDecodingTests: MereRunCoreTestCase { ) } + func testQ35DenseMTPRequiresExplicitOptIn() { + XCTAssertFalse( + Q35Generator.shouldSpeculate( + promptTokenCount: 32, + maxContextTokens: 262_144, + defaultMinimumPromptTokens: 0, + enabledByDefault: false, + environment: [:] + ) + ) + XCTAssertTrue( + Q35Generator.shouldSpeculate( + promptTokenCount: 32, + maxContextTokens: 262_144, + defaultMinimumPromptTokens: 0, + enabledByDefault: false, + environment: ["MERERUN_Q35_MTP_SPECULATION": "1"] + ) + ) + } + func testQ35MTPBlockSizeUsesEnvironmentClamp() { XCTAssertEqual(Q35Generator.mtpBlockSize(environment: [:]), 4) XCTAssertEqual(Q35Generator.mtpBlockSize(environment: ["MERERUN_Q35_MTP_BLOCK_SIZE": "1"]), 4) @@ -506,21 +713,38 @@ final class Q35ConfigDecodingTests: MereRunCoreTestCase { XCTAssertEqual(config.textConfig.numExperts, 256) } - func testQ35ConfigAllowsDenseQwen35VisionLayout() throws { + func testQ38PublishedDenseVisionConfigurationDecodes() throws { var configObject = makeBaseConfig() configObject["model_type"] = "qwen3_5" configObject["architectures"] = ["Qwen3_5ForConditionalGeneration"] - configObject["eos_token_id"] = 248_046 + configObject.removeValue(forKey: "eos_token_id") configObject["image_token_id"] = 248_056 + configObject["video_token_id"] = 248_057 configObject["vision_start_token_id"] = 248_053 configObject["vision_end_token_id"] = 248_054 if var textConfig = configObject["text_config"] as? [String: Any] { textConfig["model_type"] = "qwen3_5_text" - textConfig["hidden_size"] = 2048 - textConfig["num_hidden_layers"] = 24 - textConfig["intermediate_size"] = 6144 - textConfig["num_attention_heads"] = 8 + textConfig["hidden_size"] = 5120 + textConfig["num_hidden_layers"] = 64 + textConfig["intermediate_size"] = 17_408 + textConfig["num_attention_heads"] = 24 + textConfig["num_key_value_heads"] = 4 textConfig["head_dim"] = 256 + textConfig["layer_types"] = (0..<64).map { ($0 + 1).isMultiple(of: 4) ? "full_attention" : "linear_attention" } + textConfig["linear_num_value_heads"] = 48 + textConfig["linear_num_key_heads"] = 16 + textConfig["linear_key_head_dim"] = 128 + textConfig["linear_value_head_dim"] = 128 + textConfig["max_position_embeddings"] = 262_144 + textConfig["vocab_size"] = 248_320 + textConfig["eos_token_id"] = 248_044 + textConfig["rope_parameters"] = [ + "mrope_interleaved": true, + "mrope_section": [11, 11, 10], + "partial_rotary_factor": 0.25, + "rope_theta": 10_000_000.0, + "rope_type": "default", + ] textConfig.removeValue(forKey: "num_experts") textConfig.removeValue(forKey: "num_experts_per_tok") textConfig.removeValue(forKey: "moe_intermediate_size") @@ -529,9 +753,11 @@ final class Q35ConfigDecodingTests: MereRunCoreTestCase { } if var visionConfig = configObject["vision_config"] as? [String: Any] { visionConfig["model_type"] = "qwen3_5" - visionConfig["hidden_size"] = 1024 - visionConfig["intermediate_size"] = 4096 - visionConfig["out_hidden_size"] = 2048 + visionConfig["depth"] = 27 + visionConfig["hidden_act"] = "gelu_pytorch_tanh" + visionConfig["hidden_size"] = 1152 + visionConfig["intermediate_size"] = 4304 + visionConfig["out_hidden_size"] = 5120 visionConfig["patch_size"] = 16 visionConfig["spatial_merge_size"] = 2 visionConfig["num_position_embeddings"] = 2304 @@ -541,15 +767,54 @@ final class Q35ConfigDecodingTests: MereRunCoreTestCase { let config = try decodeConfig(configObject) XCTAssertEqual(config.modelType, "qwen3_5") - XCTAssertEqual(config.eosTokenIds, [248_046]) + XCTAssertEqual(config.eosTokenIds, [248_044]) XCTAssertEqual(config.imageTokenId, 248_056) XCTAssertFalse(config.textConfig.usesMoE) XCTAssertEqual(config.textConfig.numExperts, 0) XCTAssertEqual(config.textConfig.numExpertsPerTok, 0) + XCTAssertEqual(config.textConfig.hiddenSize, 5120) + XCTAssertEqual(config.textConfig.numHiddenLayers, 64) + XCTAssertEqual(config.textConfig.intermediateSize, 17_408) + XCTAssertEqual(config.textConfig.maxPositionEmbeddings, 262_144) + XCTAssertEqual(config.visionConfig?.depth, 27) + XCTAssertEqual(config.visionConfig?.hiddenSize, 1152) + XCTAssertEqual(config.visionConfig?.outHiddenSize, 5120) XCTAssertEqual(config.visionConfig?.patchSize, 16) XCTAssertEqual(config.visionConfig?.spatialMergeSize, 2) } + func testQ38GenerationConfigDecodesAllPublishedStopTokens() throws { + let data = Data(#"{"eos_token_id":[248046,248044]}"#.utf8) + + let config = try JSONDecoder().decode(Q35GenerationConfig.self, from: data) + + XCTAssertEqual(config.eosTokenIds, [248_046, 248_044]) + } + + func testQ38ResourceProfileUsesPublishedContextAndVisionBounds() throws { + let profile = try XCTUnwrap(Q35Resources.profile(for: Q35Resources.q38TwentySevenBModelId)) + let bounds = Q35Resources.visionPixelBounds(forModelId: profile.modelId) + + XCTAssertEqual(profile.upstreamRepoId, Q35Resources.q38TwentySevenBUpstreamRepoId) + XCTAssertEqual(profile.upstreamRevision, Q35Resources.q38TwentySevenBUpstreamRevision) + XCTAssertEqual(Q35Resources.defaultContextLength(forModelId: profile.modelId), 262_144) + XCTAssertEqual(bounds.minimum, 65_536) + XCTAssertEqual(bounds.maximum, 16_777_216) + } + + func testQ38FourBitResourceProfileKeepsPublishedRuntimeBounds() throws { + let profile = try XCTUnwrap( + Q35Resources.profile(for: Q35Resources.q38TwentySevenB4BitModelId) + ) + let bounds = Q35Resources.visionPixelBounds(forModelId: profile.modelId) + + XCTAssertEqual(profile.upstreamRepoId, Q35Resources.q38TwentySevenB4BitUpstreamRepoId) + XCTAssertEqual(profile.upstreamRevision, Q35Resources.q38TwentySevenB4BitUpstreamRevision) + XCTAssertEqual(Q35Resources.defaultContextLength(forModelId: profile.modelId), 262_144) + XCTAssertEqual(bounds.minimum, 65_536) + XCTAssertEqual(bounds.maximum, 16_777_216) + } + func testQ35ConfigAllowsOrnithOptiQQuantizationMetadata() throws { var configObject = makeBaseConfig() configObject["model_type"] = "qwen3_5" diff --git a/docs/cli.md b/docs/cli.md index 64b851a9..f4a2e7fb 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -230,7 +230,7 @@ are: `image-hidream-o1`, `image-hidream-o1-dev`, `image-krea2-raw`, `image-krea2-turbo`, `image-ideogram4-sdnq-uint4` -- Text chat: `text-chat-gemma4`, `text-chat-mebot`, `text-chat-psi-agent`, `text-chat-q36-nano`, `text-chat-lfm25-2.6b-4bit`, `text-chat-lfm25-a1b-8bit`, `vision-chat-lfm25-3b-8bit` +- Text chat: `text-chat-gemma4`, `text-chat-mebot`, `text-chat-psi-agent`, `text-chat-q36-nano`, `vision-chat-q38-27b`, `vision-chat-q38-27b-4bit`, `text-chat-lfm25-2.6b-4bit`, `text-chat-lfm25-a1b-8bit`, `vision-chat-lfm25-3b-8bit` - Text code / agents: `text-agent-qwen35-9b`, `text-agent-ornith-9b`, `text-agent-ornith-35b-mlx`, `text-agent-ornith-35b`, `text-code-north-mini`, `text-code-qwen3` - Text embed: `text-embed-qwen3-0.6b` - Text anonymize: `text-anonymize-privacy-filter` @@ -790,7 +790,7 @@ Key options: ### `mere.run text chat` -Run local text chat with the Gemma 4, Laguna 2.1, Inkling-Small, Qwen3.6/Bonsai, +Run local text chat with the Gemma 4, Laguna 2.1, Inkling-Small, Qwen3.6/Qwen3.8/Bonsai, LFM2, or Psi family. ```bash @@ -804,14 +804,14 @@ Key options: - `--model`: canonical model id - `--model-root`: explicit local model root - `--max-tokens` -- `--context-size`: maximum prompt plus generation context. Bonsai 27B uses - its published 262,144-token limit by default. Inkling-Small advertises +- `--context-size`: maximum prompt plus generation context. Qwen3.8 and Bonsai + 27B use their published 262,144-token limit by default. Inkling-Small advertises 1,048,576 tokens but uses a 32,768-token operational default because KV residency grows with context. - `--temperature`: defaults to 0.7, or the model's published value where one - exists (Bonsai: 0.7; Ornith lanes: 1.0) -- `--top-p`: defaults to 0.9, or the model's published value (Bonsai/Ornith: 0.95) -- `--top-k`: defaults to no cutoff, or the model's published value (Bonsai/Ornith: 20) + exists (Bonsai: 0.7; Qwen3.8 and Ornith lanes: 1.0) +- `--top-p`: defaults to 0.9, or the model's published value (Qwen3.8/Bonsai/Ornith: 0.95) +- `--top-k`: defaults to no cutoff, or the model's published value (Qwen3.8/Bonsai/Ornith: 20) - `--min-p`: relative probability floor from 0 through 1; `0` disables it. For example, `0.05` removes tokens below 5% of the leading token's probability. It does not change greedy generation. @@ -822,8 +822,8 @@ Key options: validates each token before streaming. - `--thinking` / `--no-thinking`: show reasoning output / disable reasoning generation. R1-style lanes (`text-agent-ornith-*`) generate with thinking - enabled by default even though the reasoning stays hidden. Bonsai 27B also - defaults to thinking-enabled generation. + enabled by default even though the reasoning stays hidden. Qwen3.8 and + Bonsai 27B also default to thinking-enabled generation. - `--stream` - `--stats`: includes user-visible `ttft_s`, decode-only `first_token_s`, separate LFM2 prefill and decode tokens/sec, and Gemma4 MTP state and @@ -845,6 +845,7 @@ Examples: ```bash swift run mere.run text chat --prompt "What is classifier-free guidance?" +swift run mere.run text chat --model vision-chat-q38-27b --image ./diagram.png --prompt "Explain this diagram." swift run mere.run text chat --model text-chat-bonsai-27b-1bit --context-size 262144 --kv-bits 4 --prompt "Plan a long-context repository review." swift run mere.run text chat --model text-chat-bonsai-27b-2bit --context-size 262144 --kv-bits 4 --prompt "Compare two repository migration plans." swift run mere.run text chat --model text-chat-inkling-small --context-size 32768 --prompt "Plan a recovery-safe repository migration." @@ -2472,12 +2473,27 @@ default suite is `humaneval-slice`, a three-task HumanEval subset covering uses the supported members of the coding comparison lane for this machine: `text-agent-ornith-9b` and `text-code-north-mini` on 32 GB Macs, with `text-code-qwen3` added on 64 GB and larger machines. Pass `--models` to force a -specific explicit comparison. +specific explicit comparison. The installed `vision-chat-q38-27b` and +`vision-chat-q38-27b-4bit` code-generation lanes are available as explicit +targets but are not added to the default comparison. The BF16 checkpoint is +55.59 GB; the 4-bit target plus official MTP shard is 19.47 GB. ```bash swift run mere.run model benchmark code \ --allow-code-execution \ --json + +swift run mere.run model benchmark code \ + --models vision-chat-q38-27b-4bit \ + --thinking \ + --max-tokens 3072 \ + --allow-code-execution \ + --json + +MERERUN_Q35_MTP_SPECULATION=1 swift run mere.run model benchmark code \ + --models vision-chat-q38-27b-4bit \ + --allow-code-execution \ + --json ``` The command prompts each model once per task, combines the generated Python with diff --git a/docs/configuration.md b/docs/configuration.md index 6fc59713..a885a122 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -461,18 +461,25 @@ Useful for locating whether decode is CPU-, schedule-, or GPU-bound. ### `MERERUN_Q35_MTP_SPECULATION` -Controls the Qwen-family MTP path used by `text-chat-q36-nano`. Set this to +Controls the Qwen-family MTP path used by `text-chat-q36-nano` and +the `vision-chat-q38-27b` BF16 and 4-bit lanes. Set this to `1`, `true`, `yes`, or `on` to force consideration when the effective context window is large enough; set it to `0`, `false`, or `no` to disable MTP. Any other -value, including unset, uses the adaptive long-context threshold. +value, including unset, uses the model-specific policy. Qwen3.8's dense head is +embedded in the BF16 checkpoint and mounted from the pinned official final shard +for `vision-chat-q38-27b-4bit`. Both are opt-in because multi-token verification +can diverge from serial greedy decode; Qwen3.6 hybrid MoE retains its adaptive +default. The `Q35` name is an internal compatibility prefix for the Qwen-family runtime; -the public managed model id is `text-chat-q36-nano`. +the public managed model ids retain their Qwen release names. ### `MERERUN_Q35_MTP_MIN_PROMPT_TOKENS` -Minimum effective prompt length before Qwen-family MTP is considered. Defaults -to `6144`, and the effective request context must also be at least this large. +Minimum effective prompt length before Qwen-family MTP is considered. Hybrid +MoE models default to `6144`; an explicitly enabled dense embedded head defaults +to `0`. The effective request context must also be at least the selected +threshold. ### `MERERUN_Q35_MTP_BLOCK_SIZE` diff --git a/docs/model-sources.md b/docs/model-sources.md index 5af435c9..7d0d64f1 100644 --- a/docs/model-sources.md +++ b/docs/model-sources.md @@ -79,6 +79,8 @@ an effective overlay; they are not a second capability catalog. | `vision-chat` | `vision-chat-muse-glimmer-30b` | | `text-chat` | `text-chat-nemotron-35-lightning` | | `text-chat` | `text-chat-q36-nano` | +| `vision-chat` | `vision-chat-q38-27b` | +| `vision-chat` | `vision-chat-q38-27b-4bit` | | `text-chat` | `text-chat-bonsai-27b-1bit` | | `text-chat` | `text-chat-bonsai-27b-2bit` | | `text-code` | `text-agent-ornith-9b` | @@ -315,6 +317,28 @@ OptiQ serving; mere.run loads that draft head when present, but only uses it for adaptive speculative decode when the effective prompt and context window are long enough. Short-context requests decode with the main chat weights. +`vision-chat-q38-27b` installs Qwen's official `Qwen/Qwen3.8-27B` BF16 +checkpoint at immutable revision +`1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0`. It is a dense 27B native +vision-language model with hybrid linear/full attention, a 262,144-token +native context window, image and video weights, and Apache-2.0 terms. mere.run +supports text and local-image prompts through the native Qwen-family runtime; +video input is not yet exposed. The selected snapshot is 55.59 GB, requires an +explicit `model pull`, and is cataloged for 64 GB unified memory minimum with +96 GB recommended. The pull retains the published generation and processor +metadata as well as the license. + +`vision-chat-q38-27b-4bit` installs +`lmstudio-community/Qwen3.8-27B-MLX-4bit` at immutable revision +`6067b15cf581666a4aecf6af3afaba4bb5efc20c`. The target uses 4-bit/group-64 +MLX affine weights and retains the same Qwen3.8 text, code, image, context, and +sampling contracts. The managed pull also mounts Qwen's official final BF16 +shard at revision `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0` under `mtp/`; +that shard contains the 15 published MTP tensors and the Apache-2.0 license. +The two pinned sources total 19.47 GB. The MTP component is loaded only when +`MERERUN_Q35_MTP_SPECULATION=1`; default decode remains target-only because +greedy speculative verification is not bit-exact with serial decode. + `text-chat-bonsai-27b-1bit` and `text-chat-bonsai-27b-2bit` install Prism ML's public `prism-ml/Bonsai-27B-mlx-1bit` and `prism-ml/Ternary-Bonsai-27B-mlx-2bit` snapshots at exact catalog revisions. diff --git a/docs/runtime/api-server.md b/docs/runtime/api-server.md index b9888ba8..256d7527 100644 --- a/docs/runtime/api-server.md +++ b/docs/runtime/api-server.md @@ -485,6 +485,12 @@ Engine compatibility: support with strict mode disabled. JSON-object mode forces thinking off and uses token-level constrained serial decoding; it does not implement `json_schema`. +- `vision-chat-q38-27b` and `vision-chat-q38-27b-4bit`: use the same native + Qwen-family serving engine for the official dense Qwen3.8 27B BF16 checkpoint + or the pinned MLX 4-bit conversion. Both accept function tools, one + local/base64 image content part per message, and structured JSON output; both + default to thinking and the published 1.0/0.95/20 sampling. Pull the 55.59 GB + BF16 lane or 19.47 GB 4-bit-plus-MTP lane explicitly before serving it. - `text-chat-bonsai-27b-1bit` and `text-chat-bonsai-27b-2bit`: use the same native Qwen-family serving engine for Prism ML's dense packed binary and ternary 27B checkpoints. They accept function tools and one local/base64 diff --git a/docs/runtime/model-management.md b/docs/runtime/model-management.md index 383694dc..7814689b 100644 --- a/docs/runtime/model-management.md +++ b/docs/runtime/model-management.md @@ -102,7 +102,7 @@ button opens the configured model store in Finder; it does not start a download. Examples: - images: `image-klein-nano`, `image-bonsai-binary`, `image-bonsai-ternary`, `image-zimage-nano`, `image-klein-max`, `image-zimage-max` -- text: `text-chat-gemma4`, `text-chat-laguna-s-2-1`, `text-chat-laguna-xs-2-1`, `text-chat-nemotron-35-lightning`, `text-chat-q36-nano`, `text-chat-bonsai-27b-1bit`, `text-chat-bonsai-27b-2bit`, `text-chat-lfm25-2.6b-4bit`, `text-chat-lfm25-a1b-8bit`, `vision-chat-lfm25-3b-8bit`, `text-agent-deepseek-v4-flash`, `text-agent-qwen35-9b`, `text-agent-ornith-9b`, `text-agent-ornith-35b-mlx`, `text-agent-ornith-35b`, `text-code-north-mini`, `text-code-qwen3`, `text-embed-qwen3-0.6b` +- text: `text-chat-gemma4`, `text-chat-laguna-s-2-1`, `text-chat-laguna-xs-2-1`, `text-chat-nemotron-35-lightning`, `text-chat-q36-nano`, `vision-chat-q38-27b`, `vision-chat-q38-27b-4bit`, `text-chat-bonsai-27b-1bit`, `text-chat-bonsai-27b-2bit`, `text-chat-lfm25-2.6b-4bit`, `text-chat-lfm25-a1b-8bit`, `vision-chat-lfm25-3b-8bit`, `text-agent-deepseek-v4-flash`, `text-agent-qwen35-9b`, `text-agent-ornith-9b`, `text-agent-ornith-35b-mlx`, `text-agent-ornith-35b`, `text-code-north-mini`, `text-code-qwen3`, `text-embed-qwen3-0.6b` - speech: `speech-tts-qwen3-nano`, `speech-asr-parakeet` - vision: `vision-ocr-lighton` - music: `music-acestep`, `music-acestep-xl-turbo`, `music-acestep-xl-turbo-lm4b`, `music-acestep-xl-sft`, `music-acestep-xl-base`, `music-acestep-lm-1.7b`, `music-acestep-lm-4b`, `music-magenta-rt2-small`, `music-magenta-rt2-base` diff --git a/docs/runtime/text.md b/docs/runtime/text.md index 49daa662..8070a8dd 100644 --- a/docs/runtime/text.md +++ b/docs/runtime/text.md @@ -43,6 +43,8 @@ help in the repository gate. - `text-chat-laguna-xs-2-1` (managed Poolside Laguna XS 2.1 33B-A3B NVFP4 target) - `text-chat-nemotron-35-lightning` (managed NVIDIA Nemotron 3.5 Lightning 30B-A3B NVFP4 target plus DSpark) - `text-chat-q36-nano` +- `vision-chat-q38-27b` (managed official Qwen3.8 27B BF16 vision-language snapshot) +- `vision-chat-q38-27b-4bit` (managed MLX 4-bit target plus pinned official MTP shard) - `text-chat-bonsai-27b-1bit` (managed packed 1-bit dense Qwen3.6 27B vision/reasoning snapshot) - `text-chat-bonsai-27b-2bit` (managed packed 2-bit ternary dense Qwen3.6 27B vision/reasoning snapshot) - `text-chat-lfm25-2.6b-4bit` (managed LiquidAI LFM2.5 2.6B dense MLX 4-bit snapshot) @@ -261,6 +263,46 @@ to a smaller 4-bit TurboQuant cache, so forcing affine 8-bit can increase that model's KV residency. `default` restores the engine/model/server default rather than promising full precision. +`vision-chat-q38-27b` is the official dense Qwen3.8 27B BF16 checkpoint. Pull +it explicitly before use because the pinned snapshot is 55.59 GB: + +```bash +swift run mere.run model pull vision-chat-q38-27b +swift run mere.run text chat \ + --model vision-chat-q38-27b \ + --image ./diagram.png \ + --prompt "Explain this diagram and verify every label." +``` + +The lane uses the published 262,144-token context, thinking default, +temperature 1.0, top-p 0.95, top-k 20, both generation stop tokens, and the +Qwen3.8 image sizing floor. The checkpoint also contains video understanding +weights, but the current native command accepts text and local images only. Its +embedded dense MTP head can be loaded from the official shards with +`MERERUN_Q35_MTP_SPECULATION=1`. This materially accelerates greedy decode, but +is experimental: BF16 multi-token verification can choose a different greedy +path from serial target decode. The default, sampled, and JSON-constrained paths +retain target-only decode. + +For the lower-residency lane, pull the separate 4-bit model ID: + +```bash +swift run mere.run model pull vision-chat-q38-27b-4bit +swift run mere.run text chat \ + --model vision-chat-q38-27b-4bit \ + --prompt "Implement a bounded async work queue in Swift." +``` + +This installs a pinned 4-bit/group-64 MLX target and Qwen's pinned final BF16 +shard under `mtp/`, totaling 19.47 GB. On the measured M4 Max coding slice, +target-only warm decode reached about 26.5 tok/s versus 8.8 tok/s for BF16; +explicit MTP reached 37.8–43.8 tok/s across the three short cases. All cases +passed. On a deterministic 24-task stride through the official HumanEval set, +target-only and explicit MTP both passed 20/24 with the same four failures, but +one failing case generated 174 tokens with MTP versus 177 target-only. MTP also +changed a thinking-mode token trajectory, so it remains explicitly opt-in and +is disabled for sampled and JSON-constrained generation. + `text-chat-bonsai-27b-1bit` and `text-chat-bonsai-27b-2bit` install the pinned 5.13 GB binary and 8.52 GB ternary Prism ML snapshots. They run packed low-bit language weights plus a dense vision tower through the native Qwen-family