Repository navigation
Add Nn::StatefulTransducerOnnxLabelScorer - #149
Merged
Merged
Conversation
SimBe195
marked this pull request as ready for review
October 2, 2025 17:56
hannah220
reviewed
Nov 12, 2025
larissakl
reviewed
Apr 21, 2026
larissakl
approved these changes
Apr 22, 2026
…orer and add model caching
…n StatefulOnnxLabelScorer and StatefulTransducerOnnxLabelScorer
Resolve the conflict with #261 in StatefulOnnxLabelScorer and port both hidden-state scorers to the StatTrackingLabelScorer interface (resetInternal, computeScoreAccessor(s)). The session runs and the state marshalling now live in OnnxHiddenStateModel, so the helper owns the session and marshalling timers. StatefulOnnxLabelScorer reports the same statistics as on master by combining them with its own marshalling timer. StatefulTransducerOnnxLabelScorer reports the same breakdown, without the initializer, which runs outside of the scoring time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
StatefulTransducerOnnxLabelScorer runs its state initializer in getInitialScoringContext, outside getScoreAccessor(s), so neither the scoring-time total nor the breakdown included it. Time it with scoringTime_ and report initializer + updater as state-update-session-time, as StatefulOnnxLabelScorer already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reorder the definitions in StatefulTransducerOnnxLabelScorer.cc and OnnxHiddenStateModel.cc to follow the declaration order in the headers. No code changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move OnnxHiddenState and OnnxHiddenStateModel from Nn/LabelScorer into src/Onnx (namespace Onnx), next to the other ONNX implementations of Nn interfaces. ScoringContext.hh no longer includes Onnx/Value.hh, so headers shared by all searches don't pull in ONNX Runtime. Register the ONNX label scorers in Nn::Module only with MODULE_ONNX. CtcPrefixLabelScorer.hh used Math::FastMatrix without including it and only got it transitively through Onnx/Value.hh; include it directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nn::Module included Onnx headers and defined createStateManager (OnnxLstmStateManager, OnnxTransformerStateManager) unconditionally, so builds with MODULE_ONNX=OFF failed to link. Put the includes, the state-manager type choice and parameter, and createStateManager under MODULE_ONNX. Its callers (StateManagedOnnxLabelScorer and Lm::OnnxRecurrentLanguageModel) are only built with ONNX enabled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
curufinwe
reviewed
Oct 28, 2026
| #ifndef CTC_PREFIX_LABEL_SCORER_HH | ||
| #define CTC_PREFIX_LABEL_SCORER_HH | ||
|
|
||
| #include <Math/FastMatrix.hh> |
Contributor
There was a problem hiding this comment.
This header uses Math::FastMatrix (ctcScores_, the CtcPrefixScoreAccessor constructor) but never included it. It only compiled because the type arrived transitively: LabelScorer.hh → DataView.hh → Onnx/Value.hh → Math/FastMatrix.hh. DataView.hh includes Onnx/Value.hh only under #ifdef MODULE_ONNX, so on master a build with -DMODULE_ONNX=OFF already fails here.
The previous state of this PR hid that, because ScoringContext.hh included Onnx/Value.hh unconditionally (for OnnxHiddenState). Now that OnnxHiddenState lives in Onnx/OnnxHiddenStateModel.hh and ScoringContext.hh is back to master's version, the transitive path is gone in ONNX-off builds. The explicit include fixes that; the ONNX-off build is green with it.
— Claude Code
curufinwe
changed the title
larissakl
requested changes
Oct 29, 2026
The class bundles three Onnx::Models (state initializer, state updater and scorer) rather than being one, so "Model" was misleading. Its state type is just a set of named ONNX values, hence OnnxHiddenState becomes OnnxNamedValues (stateValueMap -> valueMap). The bundle methods initialHiddenState/updatedHiddenStates become initialState/updatedStates, and the scorer members hiddenStateModel_ become initUpdateScoreBundle_. No functional change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The hidden state only depends on the label sequence, not on the step, so `stateCache_` now uses `LabelSeqHash`/`LabelSeqEq`. The same label sequence reached at different steps no longer reruns the state updater, and already finalized contexts are no longer inserted into the cache. Since `FIFOCache::put` doesn't refresh an existing key, a cached state could be evicted between being looked up and being read. States needed within one `computeScoreAccessors` call are therefore collected in a local map when they are looked up or computed, and all later reads use that map. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Remove the outdated comment on `getInitialScoringContext` - Pass `requiresFinalize` directly to the scoring context constructor - Return the minimum active step from `getMinActiveInputIndex` so that the input buffer is trimmed; only the input of the current step is read Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
larissakl
approved these changes
Oct 30, 2026
curufinwe
approved these changes
Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a new label scorer class
StatefulTransducerOnnxLabelScorer. This is similar to theStatefulOnnxLabelScorerbut hidden-state and encoder features are decoupled. Here, the state initializer and state updater only take tokens and no encoder features. Instead, the scorer receives the encoder state at the current time step and the hidden state and is supposed to combine the two.This can be used for transducer models with unlimited context.
Depends on #144.