Fix ModelPatcher support in tiled latent upscaler - #18
Open
jjdejong wants to merge 6 commits into
Open
Conversation
…very Adds a fourth reference_source, "reference_prefix", to LTXLikenessAnchor, which draws the identity reference from the token prefix injected by LTX Reference Enable rather than from a guide's appended frames or latent frame 0. LTXLatentAnchorAware gains matching prefix awareness: it strips the reference prefix before gridding and re-concatenates it untouched, so reference tokens are never pulled toward themselves. Three fixes to the dim-discovery path in LTXLikenessAnchor: 1. Locate the video latent explicitly instead of guessing by rank. The pre-hook scanned forward's args/kwargs for "the first 5D tensor", commented as "typically the latent input". It never was: LTX-AV passes x as a video+audio container (see separate_audio_and_video_latents, which indexes x[0]/x[1]), not a bare tensor, so isinstance skipped it. The only 5D tensor usually present is denoise_mask (B,1,F,H,W), whose F/H/W coincidentally match the latent's -- masking the bug on i2v graphs while silently no-op'ing on T2V graphs that pass no mask. _extract_video_latent now resolves x (keyword or first positional), unwraps a .tensors wrapper or plain list, and excludes mask tensors from the last-resort scan. 2. Report early exits. The hook had eight bail-out paths that returned the input unmodified with no output even under debug, making a bad bbox, a spatial mismatch and fix 1's silent failure indistinguishable from a working run. Each now explains itself once per reason, and a one-shot HOOK ACTIVE line announces the first successful fire, in the same format LTXLatentAnchorAware already used. 3. Skip a full-activation clone that was overwritten in its entirety. In reference_prefix mode the generated range covers every frame, so grid.clone() was discarded wholesale -- about 70MB per block, ~3.5GB of transient alloc/free per forward across 48 blocks at LTX-AV sizes. Verified bit-identical output in all three source modes. Note: latent_anchor_aware.py's own _capture_5d has the same rank-guessing flaw described in fix 1 and is NOT addressed here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_capture_5d scanned forward's args/kwargs for the first rank-5 tensor, which never matched the video latent: LTX-AV passes x as a [video, audio] container rather than a Tensor. What it actually picked up was denoise_mask (B,1,F,H,W), whose F/H/W happen to equal the latent's, so i2v graphs worked by coincidence while graphs without a mask captured nothing and left the anchor silently inactive. Reuse _extract_video_latent from latent_likeness_anchor, which unwraps the container explicitly and excludes mask tensors, and report the failure under debug instead of going quiet. Same fix as 6d2d85f applied to the sibling node. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Registration was gated on a sentinel attribute living on the backbone, which model.clone() shares by reference. From the second run onward in a single ComfyUI process the gate was already set, so the new pre-hook was never attached and the first run's closure stayed live, writing into a state dict nothing else could read. The effect was that captured_latent_shape and current_sigma stayed None for the rest of the process: latent_frame_0 and reference_prefix silently returned unmodified output from the second prompt onward, and the skip_when_sigma_above gate stopped working in every mode including guide. Only the first prompt after a restart ever worked. Track the handle, remove any prior registration, and re-register unconditionally -- the same lifecycle the attn1 hooks already use, and that LTXLatentAnchorAware already applies to both hook types. The bypass path now detaches the pre-hook too instead of only dropping the sentinel, which previously left it attached and firing. Found via the early-exit diagnostics added in 6d2d85f, which reported "video latent shape was never captured" on a second run; before that the failure was silent and looked like success. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #14.\n\nComfyUI now provides LATENT_UPSCALE_MODEL as a ModelPatcher. The tiled node treated it as a raw torch module, causing AttributeError when calling parameters().\n\nThis change loads the patcher through model_management, runs its wrapped model, and unloads with the supported ModelPatcher API.\n\nValidation: py_compile and isolated node import passed.