Load model from checkpoints/kokoro-v0_19.onnx failed:This is an invalid model

[7jrxt42BxFZo4iAnN4CX]

Libs:
onnx: 1.17.0-1
onnxruntime: 1.20.2-2
ort: 2.0.0-rc.4

Error:
576 messages "Schema error: Trying to register schema with name"

......

Schema error: Trying to register schema with name TreeEnsembleClassifier (domain: ai.onnx.ml version: 3) from file /usr/src/debug/onnx/onnx/onnx/defs/traditionalml/old.cc line 175, but it is already registered from file /usr/src/debug/onnx/onnx/onnx/defs/traditionalml/old.cc line 175

Schema error: Trying to register schema with name TreeEnsembleRegressor (domain: ai.onnx.ml version: 3) from file /usr/src/debug/onnx/onnx/onnx/defs/traditionalml/old.cc line 422, but it is already registered from file /usr/src/debug/onnx/onnx/onnx/defs/traditionalml/old.cc line 422

Schema error: Trying to register schema with name LabelEncoder (domain: ai.onnx.ml version: 4) from file /usr/src/debug/onnx/onnx/onnx/defs/traditionalml/defs.cc line 332, but it is already registered from file /usr/src/debug/onnx/onnx/onnx/defs/traditionalml/defs.cc line 332

Schema error: Trying to register schema with name TreeEnsemble (domain: ai.onnx.ml version: 5) from file /usr/src/debug/onnx/onnx/onnx/defs/traditionalml/defs.cc line 1017, but it is already registered from file /usr/src/debug/onnx/onnx/onnx/defs/traditionalml/defs.cc line 1017

Schema error: Trying to register schema with name TreeEnsembleRegressor (domain: ai.onnx.ml version: 5) from file /usr/src/debug/onnx/onnx/onnx/defs/traditionalml/defs.cc line 917, but it is already registered from file /usr/src/debug/onnx/onnx/onnx/defs/traditionalml/defs.cc line 917

Schema error: Trying to register schema with name TreeEnsembleClassifier (domain: ai.onnx.ml version: 5) from file /usr/src/debug/onnx/onnx/onnx/defs/traditionalml/defs.cc line 800, but it is already registered from file /usr/src/debug/onnx/onnx/onnx/defs/traditionalml/defs.cc line 800

thread 'main' panicked at /Kokoros/kokoros/src/tts/koko.rs:68:18:
Failed to create Kokoro TTS model: "Failed to commit from file: Load model from checkpoints/kokoro-v0_19.onnx failed:This is an invalid model. In Node, ("Identity_1316", Identity, "", -1) : ("onnx::MatMul_7102": tensor(float),) -> ("onnx::MatMul_7245",) , Error /usr/src/debug/onnx/onnx/onnx/defs/schema.cc:1116: SchemasRegisterer: Assertion dbg_registered_schema_count == DbgOperatorSetTracker::Instance().GetCount() failed: 4 schema were exposed from operator sets and automatically placed into the static registry. 696 were expected based on calls to registration macros. Operator set functions may need to be updated."
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::result::unwrap_failed
3: koko::main::{{closure}}
4: tokio::runtime::context::runtime::enter_runtime
5: koko::main

Solution:
Downgrade onnxruntime to 1.19.2-4

yay downgrade
sudo downgrade onnxruntime

[lucasjinreal]

Hello, you mean it need downgrade onnxruntime?

We might need support newest kokoro onnx model

[7jrxt42BxFZo4iAnN4CX]

Yes, on onnxruntime version 1.20.2-2 does not work

On February 16th a new version (1.20.2) was released, which broke everything.
https://gitlab.archlinux.org/archlinux/packaging/packages/onnxruntime/-/commits/main

[lucasjinreal]

Anybody knows how can we fix this be reexporting onnx file? If 2.0 of onnxruntime caused this, it might be issues for many of users

[7jrxt42BxFZo4iAnN4CX]

Hurried with the message that everything works. I was only able to compile, but now the audio file is damaged.

I have no idea what happened at all, everything worked two weeks ago.

[7jrxt42BxFZo4iAnN4CX]

After commit 272b7b0 the error persists:

Schema error: Trying to register schema with name TreeEnsembleClassifier (domain: ai.onnx.ml version: 5) from file /usr/src/debug/onnx/onnx/onnx/defs/traditionalml/defs.cc line 800, but it is already registered from file /usr/src/debug/onnx/onnx/onnx/defs/traditionalml/defs.cc line 800

thread 'main' panicked at /home/f2/Documents/Shared/ProL/Rust/ai/Kokoros/kokoros/src/tts/koko.rs:69:18:
Failed to create Kokoro TTS model: "Failed to commit from file: Load model from checkpoints/kokoro-v1.0.onnx failed:This is an invalid model. In Node, ("/encoder/bert/Shape_1", Shape, "", -1) : ("tokens": tensor(int64),) -> ("/encoder/bert/Shape_1_output_0": tensor(int64),) , Error /usr/src/debug/onnx/onnx/onnx/defs/schema.cc:1116: SchemasRegisterer: Assertion dbg_registered_schema_count == DbgOperatorSetTracker::Instance().GetCount() failed: 4 schema were exposed from operator sets and automatically placed into the static registry. 696 were expected based on calls to registration macros. Operator set functions may need to be updated."

extra/onnxruntime-rocm 1.20.2-3 (20.7 MiB 1.5 GiB) (Installed)
extra/onnx 1:1.17.0-2 (1.0 MiB 4.7 MiB) (Installed)

[7jrxt42BxFZo4iAnN4CX]

on onnxruntime version 1.19.2(commit 272b7b0) the file is obtained with a small artifact at the beginning: https://github.com/7jrxt42BxFZo4iAnN4CX/Kokoros/blob/wav_artifact_onnxruntime_1_19_2/sky-says-hello.wav

curl -X POST http://localhost:3000/v1/audio/speech \
                -H "Content-Type: application/json" \
                -d '{
              "model": "anything can go here",
              "input": "Hello, this is a test of the Kokoro TTS system!",
              "voice": "af_sky", 
              "response_format": "wav"
            }' --output sky-says-hello.wav

[lucasjinreal]

Oh, is this on Kokoro 1.0 model? The beginning artifact seems a known issue. We might need to have a check on sending tokens and compare whether the output is the same as the python side or not.

[7jrxt42BxFZo4iAnN4CX]

yes, Kokoro 1.0 model (commit 272b7b0)
I don't know how to fix this

[7jrxt42BxFZo4iAnN4CX]

Oh, is this on Kokoro 1.0 model? The beginning artifact seems a known issue. We might need to have a check on sending tokens and compare whether the output is the same as the python side or not.

Should i wait for a fix?