Performance related question

[MaraScott]

Hi,

thx for the good work.
I tried the inference_offline on a 32 sec video and it renders perfectly.

I tried inference_online use_trt = False cause unet_work.engine which is h100 based does t work for me.
I noticed I need to trigger torch2trt.py to replace the provided unet_work.engine.
aside of that, how can I make the 25fps online version works smoothly ?
What should I look into to increase performance ?

[MaraScott]

Running torch2trt, I got this and it is stuck on [V] Compiler backend is used during engine build. for more than 1 hour.

(personalive) PS C:\Users\<myuser>\Program_Files\PersonaLive> python torch2trt.py
A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
Warning: Accessing the `severity` property of G_LOGGER is deprecated and will be removed in v0.50.0. Use `module_severity` instead
Pipeline:  20%|██████████▍                                         | 2/10 [00:04<00:18,  2.28s/step, Load UNet + Temporal]C:\Users\<myuser>\miniconda3\envs\personalive\lib\site-packages\torch\_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()   
  return self.fget.__get__(instance, owner)()
Pipeline:  50%|████████████████████████▌                        | 5/10 [00:50<01:08, 13.66s/step, Export ONNX (if needed)]Dynamic: d00   | Base(1x): 4096  | Range: 4096 ~ 16384 | Opt: 8192
Dynamic: d01   | Base(1x): 4096  | Range: 4096 ~ 16384 | Opt: 8192
Dynamic: d10   | Base(1x): 1024  | Range: 1024 ~ 4096 | Opt: 2048
Dynamic: d11   | Base(1x): 1024  | Range: 1024 ~ 4096 | Opt: 2048
Dynamic: d20   | Base(1x): 256   | Range: 256 ~ 1024 | Opt: 512
Dynamic: d21   | Base(1x): 256   | Range: 256 ~ 1024 | Opt: 512
Dynamic: m     | Base(1x): 64    | Range: 64 ~ 256 | Opt: 128
Dynamic: u10   | Base(1x): 256   | Range: 256 ~ 1024 | Opt: 512
Dynamic: u11   | Base(1x): 256   | Range: 256 ~ 1024 | Opt: 512
Dynamic: u12   | Base(1x): 256   | Range: 256 ~ 1024 | Opt: 512
Dynamic: u20   | Base(1x): 1024  | Range: 1024 ~ 4096 | Opt: 2048
Dynamic: u21   | Base(1x): 1024  | Range: 1024 ~ 4096 | Opt: 2048
Dynamic: u22   | Base(1x): 1024  | Range: 1024 ~ 4096 | Opt: 2048
Dynamic: u30   | Base(1x): 4096  | Range: 4096 ~ 16384 | Opt: 8192
Dynamic: u31   | Base(1x): 4096  | Range: 4096 ~ 16384 | Opt: 8192
Dynamic: u32   | Base(1x): 4096  | Range: 4096 ~ 16384 | Opt: 8192
Pipeline:  70%|█████████████████████████████████████████▎                 | 7/10 [00:50<00:15,  5.11s/step, Optimize ONNX]Saving to ./pretrained_weights/onnx/unet_opt/unet_opt.onnx...
Optimization done.
Pipeline:  80%|████████████████████████████████████████▊          | 8/10 [01:50<00:35, 17.75s/step, Build TensorRT engine][V] Loaded Module: tensorrt | Version: 10.14.1.48.post1 | Path: ['C:\\Users\\<myuser>\\miniconda3\\envs\\personalive\\Lib\\site-packages\\tensorrt']
[V] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 12618, GPU 4906 (MiB)
[V] ----------------------------------------------------------------
[V] Input filename:   ./pretrained_weights/onnx/unet_opt/unet_opt.onnx
[V] ONNX IR version:  0.0.8
[V] Opset version:    17
[V] Producer name:    pytorch
[V] Producer version: 2.1.0
[V] Domain:
[V] Model version:    0
[V] Doc string:
[V] ----------------------------------------------------------------
[I] TF32 is disabled by default. Turn on TF32 for better performance with minor accuracy differences.
[V] Setting TensorRT Optimization Profiles
[V] Input tensor: sample (dtype=DataType.HALF, shape=(1, 4, 16, -1, -1)) | Setting input tensor shapes to: (min=(1, 4, 16, 64, 64), opt=(1, 4, 16, 64, 64), max=(1, 4, 16, 64, 64))
[V] Input tensor: encoder_hidden_states (dtype=DataType.HALF, shape=(1, 1, 768)) | Setting input tensor shapes to: (min=(1, 1, 768), opt=(1, 1, 768), max=(1, 1, 768))
[V] Input tensor: motion_hidden_states (dtype=DataType.HALF, shape=(1, 12, 32, 16)) | Setting input tensor shapes to: (min=(1, 12, 32, 16), opt=(1, 12, 32, 16), max=(1, 12, 32, 16))
[V] Input tensor: motion (dtype=DataType.HALF, shape=(1, 3, 4, 224, 224)) | Setting input tensor shapes to: (min=(1, 3, 4, 224, 224), opt=(1, 3, 4, 224, 224), max=(1, 3, 4, 224, 224))
[V] Input tensor: pose_cond_fea (dtype=DataType.HALF, shape=(1, 320, 12, -1, -1)) | Setting input tensor shapes to: (min=(1, 320, 12, 64, 64), opt=(1, 320, 12, 64, 64), max=(1, 320, 12, 64, 64))
[V] Input tensor: pose (dtype=DataType.HALF, shape=(1, 3, 4, -1, -1)) | Setting input tensor shapes to: (min=(1, 3, 4, 512, 512), opt=(1, 3, 4, 512, 512), max=(1, 3, 4, 512, 512))
[V] Input tensor: new_noise (dtype=DataType.HALF, shape=(1, 4, 4, -1, -1)) | Setting input tensor shapes to: (min=(1, 4, 4, 64, 64), opt=(1, 4, 4, 64, 64), max=(1, 4, 4, 64, 64))
[V] Input tensor: d00 (dtype=DataType.HALF, shape=(1, -1, 320)) | Setting input tensor shapes to: (min=(1, 4096, 320), opt=(1, 8192, 320), max=(1, 16384, 320))
[V] Input tensor: d01 (dtype=DataType.HALF, shape=(1, -1, 320)) | Setting input tensor shapes to: (min=(1, 4096, 320), opt=(1, 8192, 320), max=(1, 16384, 320))
[V] Input tensor: d10 (dtype=DataType.HALF, shape=(1, -1, 640)) | Setting input tensor shapes to: (min=(1, 1024, 640), opt=(1, 2048, 640), max=(1, 4096, 640))
[V] Input tensor: d11 (dtype=DataType.HALF, shape=(1, -1, 640)) | Setting input tensor shapes to: (min=(1, 1024, 640), opt=(1, 2048, 640), max=(1, 4096, 640))
[V] Input tensor: d20 (dtype=DataType.HALF, shape=(1, -1, 1280)) | Setting input tensor shapes to: (min=(1, 256, 1280), opt=(1, 512, 1280), max=(1, 1024, 1280))
[V] Input tensor: d21 (dtype=DataType.HALF, shape=(1, -1, 1280)) | Setting input tensor shapes to: (min=(1, 256, 1280), opt=(1, 512, 1280), max=(1, 1024, 1280))
[V] Input tensor: m (dtype=DataType.HALF, shape=(1, -1, 1280)) | Setting input tensor shapes to: (min=(1, 64, 1280), opt=(1, 128, 1280), max=(1, 256, 1280))
[V] Input tensor: u10 (dtype=DataType.HALF, shape=(1, -1, 1280)) | Setting input tensor shapes to: (min=(1, 256, 1280), opt=(1, 512, 1280), max=(1, 1024, 1280))
[V] Input tensor: u11 (dtype=DataType.HALF, shape=(1, -1, 1280)) | Setting input tensor shapes to: (min=(1, 256, 1280), opt=(1, 512, 1280), max=(1, 1024, 1280))
[V] Input tensor: u12 (dtype=DataType.HALF, shape=(1, -1, 1280)) | Setting input tensor shapes to: (min=(1, 256, 1280), opt=(1, 512, 1280), max=(1, 1024, 1280))
[V] Input tensor: u20 (dtype=DataType.HALF, shape=(1, -1, 640)) | Setting input tensor shapes to: (min=(1, 1024, 640), opt=(1, 2048, 640), max=(1, 4096, 640))
[V] Input tensor: u21 (dtype=DataType.HALF, shape=(1, -1, 640)) | Setting input tensor shapes to: (min=(1, 1024, 640), opt=(1, 2048, 640), max=(1, 4096, 640))
[V] Input tensor: u22 (dtype=DataType.HALF, shape=(1, -1, 640)) | Setting input tensor shapes to: (min=(1, 1024, 640), opt=(1, 2048, 640), max=(1, 4096, 640))
[V] Input tensor: u30 (dtype=DataType.HALF, shape=(1, -1, 320)) | Setting input tensor shapes to: (min=(1, 4096, 320), opt=(1, 8192, 320), max=(1, 16384, 320))
[V] Input tensor: u31 (dtype=DataType.HALF, shape=(1, -1, 320)) | Setting input tensor shapes to: (min=(1, 4096, 320), opt=(1, 8192, 320), max=(1, 16384, 320))
[V] Input tensor: u32 (dtype=DataType.HALF, shape=(1, -1, 320)) | Setting input tensor shapes to: (min=(1, 4096, 320), opt=(1, 8192, 320), max=(1, 16384, 320))
[I] Configuring with profiles:[
        Profile 0:
            {sample [min=(1, 4, 16, 64, 64), opt=(1, 4, 16, 64, 64), max=(1, 4, 16, 64, 64)],
             encoder_hidden_states [min=(1, 1, 768), opt=(1, 1, 768), max=(1, 1, 768)],
             motion_hidden_states [min=(1, 12, 32, 16), opt=(1, 12, 32, 16), max=(1, 12, 32, 16)],
             motion [min=(1, 3, 4, 224, 224), opt=(1, 3, 4, 224, 224), max=(1, 3, 4, 224, 224)],
             pose_cond_fea [min=(1, 320, 12, 64, 64), opt=(1, 320, 12, 64, 64), max=(1, 320, 12, 64, 64)],
             pose [min=(1, 3, 4, 512, 512), opt=(1, 3, 4, 512, 512), max=(1, 3, 4, 512, 512)],
             new_noise [min=(1, 4, 4, 64, 64), opt=(1, 4, 4, 64, 64), max=(1, 4, 4, 64, 64)],
             d00 [min=(1, 4096, 320), opt=(1, 8192, 320), max=(1, 16384, 320)],
             d01 [min=(1, 4096, 320), opt=(1, 8192, 320), max=(1, 16384, 320)],
             d10 [min=(1, 1024, 640), opt=(1, 2048, 640), max=(1, 4096, 640)],
             d11 [min=(1, 1024, 640), opt=(1, 2048, 640), max=(1, 4096, 640)],
             d20 [min=(1, 256, 1280), opt=(1, 512, 1280), max=(1, 1024, 1280)],
             d21 [min=(1, 256, 1280), opt=(1, 512, 1280), max=(1, 1024, 1280)],
             m [min=(1, 64, 1280), opt=(1, 128, 1280), max=(1, 256, 1280)],
             u10 [min=(1, 256, 1280), opt=(1, 512, 1280), max=(1, 1024, 1280)],
             u11 [min=(1, 256, 1280), opt=(1, 512, 1280), max=(1, 1024, 1280)],
             u12 [min=(1, 256, 1280), opt=(1, 512, 1280), max=(1, 1024, 1280)],
             u20 [min=(1, 1024, 640), opt=(1, 2048, 640), max=(1, 4096, 640)],
             u21 [min=(1, 1024, 640), opt=(1, 2048, 640), max=(1, 4096, 640)],
             u22 [min=(1, 1024, 640), opt=(1, 2048, 640), max=(1, 4096, 640)],
             u30 [min=(1, 4096, 320), opt=(1, 8192, 320), max=(1, 16384, 320)],
             u31 [min=(1, 4096, 320), opt=(1, 8192, 320), max=(1, 16384, 320)],
             u32 [min=(1, 4096, 320), opt=(1, 8192, 320), max=(1, 16384, 320)]}
    ]
[W] profileSharing0806 is on by default in TensorRT 10.0. This flag is deprecated and has no effect.
[I] Building engine with configuration:
    Flags                  | [FP16]
    Engine Capability      | EngineCapability.STANDARD
    Memory Pools           | [WORKSPACE: 24575.69 MiB, TACTIC_DRAM: 24575.69 MiB, TACTIC_SHARED_MEMORY: 1024.00 MiB]       
    Tactic Sources         | [EDGE_MASK_CONVOLUTIONS, JIT_CONVOLUTIONS]
    Profiling Verbosity    | ProfilingVerbosity.DETAILED
    Preview Features       | [PROFILE_SHARING_0806]
[V] [MemUsageChange] Init builder kernel library: CPU +216, GPU +2, now: CPU 13508, GPU 4908 (MiB)
[V] Global timing cache in use. Profiling results in this builder pass will be stored.
[V] Compiler backend is used during engine build.

[huai-chang]

Hi, from the logs, it looks like you are running this on Windows?

For reference, our tests were conducted on Linux environments with A6000 and H100 GPUs, where this building step usually finishes in around 20 minutes.

I am not familiar with TensorRT's performance on Windows. Engine building can sometimes be very slow depending on the hardware and OS. You might want to check your Task Manager to see if the CPU/GPU is still active. If it is, it might just need more time, but I can't say for sure. 😢

[MaraScott]

Hi, not sure if this make sense but it is still running here #11 I will update you when I know how long it took

[huai-chang]

Thanks for the update!

Once it finishes (hopefully successfully! 🤞), if you could create a torch2trt_win.py and submit it as a PR, that would be amazing. I'd love to integrate it to support the Windows community better. 🙏

[Jeremy8776]

I've been playing with it on Windows [RTX 3090 FE] I found two issues preventing the TensorRT engine from running correctly.

The script sets the TensorRT profile to 512x512 but exports the ONNX model at 256x256 resulting in shape mismatch. Making it the same seems to resolve that.

Right now i get CUDA: Invalid argument error, at torch.norm. I tried upcasting to float32 but thats hitting the same so will try manual implementation to bypass the buggy kernal.

[huai-chang]

I've been playing with it on Windows [RTX 3090 FE] I found two issues preventing the TensorRT engine from running correctly.

The script sets the TensorRT profile to 512x512 but exports the ONNX model at 256x256 resulting in shape mismatch. Making it the same seems to resolve that.

Right now i get CUDA: Invalid argument error, at torch.norm. I tried upcasting to float32 but thats hitting the same so will try manual implementation to bypass the buggy kernal.

I actually tried exporting the ONNX model at 512x512 previously. However, this caused the ONNX model size to explode (>40GB), making it impossible to save.

Therefore, I opted to export it at 256x256 but configured the dimensions to be dynamic. This design is intended to allow the engine to handle variable resolutions (like 512x512) during inference despite the lower export resolution.

PersonaLive/src/modeling/framed_models.py

Lines 92 to 115 in 57b927f

	def get_dynamic_axes(self):
	dynamic_axes = {
	"sample": {3:"h_64", 4:"w_64"},
	"pose_cond_fea": {3:"h_64", 4:"w_64"},
	"pose": {3:"h_512", 4:"h_512"},
	"new_noise": {3: "h_64", 4: "w_64"},
	"d00" : {1: "len_4096"},
	"d01" : {1: "len_4096"},
	"u30" : {1: "len_4096"},
	"u31" : {1: "len_4096"},
	"u32" : {1: "len_4096"},
	"d10" : {1: "len_1024"},
	"d11" : {1: "len_1024"},
	"u20" : {1: "len_1024"},
	"u21" : {1: "len_1024"},
	"u22" : {1: "len_1024"},
	"d20" : {1: "len_256"},
	"d21" : {1: "len_256"},
	"u10" : {1: "len_256"},
	"u11" : {1: "len_256"},
	"u12" : {1: "len_256"},
	"m" : {1: "len_64"},
	}
	return dynamic_axes

[Jeremy8776]

yeah I saw that but on Windows I was still hitting the mismatch errors.

The main one im getting now is the Runtime Error

[huai-chang]

yeah I saw that but on Windows I was still hitting the mismatch errors.

The main one im getting now is the Runtime Error

Maybe you can refer to this commit: a61c239

I'm really not familiar with Windows-specific issues, sorry about that. 😢

[huai-chang]

Hi,

thx for the good work. I tried the inference_offline on a 32 sec video and it renders perfectly.

I tried inference_online use_trt = False cause unet_work.engine which is h100 based does t work for me. I noticed I need to trigger torch2trt.py to replace the provided unet_work.engine. aside of that, how can I make the 25fps online version works smoothly ? What should I look into to increase performance ?

By the way, the 25fps mentioned in the demo refers to the frequency at which the frontend transmits camera images to the backend, rather than the FPS of the generated video. 😂

Also, due to various latencies such as data transmission between the frontend and backend, the current online demo cannot reach the speed of the offline demo.

[Jeremy8776]

@MaraScott

How are you going about CUDA tensors not being shared between processes on windows. Im stuck.

[MaraScott]

Thanks for the update!

Once it finishes (hopefully successfully! 🤞), if you could create a torch2trt_win.py and submit it as a PR, that would be amazing. I'd love to integrate it to support the Windows community better. 🙏

this is done #11

it worked here - meaning I ve successfuly created a unet_work.engine with my Quatro RTX 6000 - it tooks 2 hours ;(

[MaraScott]

but now when I trigger the $env:CUDA_LAUNCH_BLOCKING="1"; $env:TORCH_SHOW_CPP_STACKTRACES="1"; $env:TORCH_CPP_LOG_LEVEL="INFO"; python inference_online.py, I have :

$env:CUDA_LAUNCH_BLOCKING="1"; $env:TORCH_SHOW_CPP_STACKTRACES="1"; $env:TORCH_CPP_LOG_LEVEL="INFO"; python inference_online.py


host: 0.0.0.0
port: 7860
reload: False
mode: default
max_queue_size: 0
timeout: 0.0
safety_checker: False
taesd: True
ssl_certfile: None
ssl_keyfile: None
debug: False
acceleration: xformers
engine_dir: engines
config_path: ./configs/prompts/personalive_online.yaml


C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\diffusers\models\dual_transformer_2d.py:20: FutureWarning: `DualTransformer2DModel` is deprecated and will be removed in version 0.29. Importing `DualTransformer2DModel` from `diffusers.models.dual_transformer_2d` is deprecated and this will be removed in a future version. Please use `from diffusers.models.transformers.dual_transformer_2d import DualTransformer2DModel`, instead.
  deprecate("DualTransformer2DModel", "0.29", deprecation_message)


host: 0.0.0.0
port: 7860
reload: False
mode: default
max_queue_size: 0
timeout: 0.0
safety_checker: False
taesd: True
ssl_certfile: None
ssl_keyfile: None
debug: False
acceleration: xformers
engine_dir: engines
config_path: ./configs/prompts/personalive_online.yaml


C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\diffusers\models\dual_transformer_2d.py:20: FutureWarning: `DualTransformer2DModel` is deprecated and will be removed in version 0.29. Importing `DualTransformer2DModel` from `diffusers.models.dual_transformer_2d` is deprecated and this will be removed in a future version. Please use `from diffusers.models.transformers.dual_transformer_2d import DualTransformer2DModel`, instead.
  deprecate("DualTransformer2DModel", "0.29", deprecation_message)
Some weights of the model checkpoint were not used when initializing UNet2DConditionModel: 
 ['conv_norm_out.weight, conv_norm_out.bias, conv_out.weight, conv_out.bias']
Loading engine from file ./pretrained_weights/tensorrt/unet_work.engine...
C:\Users\<username>\Program_Files\PersonaLive\inference_online.py:239: DeprecationWarning: 
        on_event is deprecated, use lifespan event handlers instead.

        Read more about it in the
        [FastAPI docs for Lifespan Events](https://fastapi.tiangolo.com/advanced/events/).

  @self.app.on_event("shutdown")
init done
INFO:     Started server process [5964]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:7860 (Press CTRL+C to quit)
INFO:     2a01:cb04:5f3:6900:c516:30a3:1d80:1ae1:0 - "GET / HTTP/1.1" 304 Not Modified
C:\Users\<username>\Program_Files\PersonaLive\inference_online.py:206: PydanticDeprecatedSince20: The `schema` method is deprecated; use `model_json_schema` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
  info_schema = pipeline.Info.schema()
C:\Users\<username>\Program_Files\PersonaLive\inference_online.py:211: PydanticDeprecatedSince20: The `schema` method is deprecated; use `model_json_schema` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
  input_params = pipeline.InputParams.schema()
INFO:     2a01:cb04:5f3:6900:c516:30a3:1d80:1ae1:0 - "GET /api/settings HTTP/1.1" 200 OK
INFO:     2a01:cb04:5f3:6900:c516:30a3:1d80:1ae1:0 - "POST /api/upload_reference_image HTTP/1.1" 200 OK
C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\diffusers\models\resnet.py:328: FutureWarning: `scale` is deprecated and will be removed in version 1.0.0. The `scale` argument is deprecated and will be ignored. Please remove it, as passing it will raise an error in the future. `scale` should directly be passed while calling the underlying pipeline component i.e., via `cross_attention_kwargs`.
  deprecate("scale", "1.0.0", deprecation_message)
C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\diffusers\models\downsampling.py:136: FutureWarning: `scale` is deprecated and will be removed in version 1.0.0. The `scale` argument is deprecated and will be ignored. Please remove it, as passing it will raise an error in the future. `scale` should directly be passed while calling the underlying pipeline component i.e., via `cross_attention_kwargs`.
  deprecate("scale", "1.0.0", deprecation_message)
C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\diffusers\models\upsampling.py:147: FutureWarning: `scale` is deprecated and will be removed in version 1.0.0. The `scale` argument is deprecated and will be ignored. Please remove it, as passing it will raise an error in the future. `scale` should directly be passed while calling the underlying pipeline component i.e., via `cross_attention_kwargs`.
  deprecate("scale", "1.0.0", deprecation_message)
INFO:     2a01:cb04:5f3:6900:c516:30a3:1d80:1ae1:0 - "WebSocket /api/ws/e88560e3-9d0f-4aaa-b5a3-3412e975e18a" [accepted]
User count: 0
New user connected: e88560e3-9d0f-4aaa-b5a3-3412e975e18a
INFO:     connection open
fuse reference done
C:\Users\<username>\Program_Files\PersonaLive\webcam\util.py:32: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at ..\torch\csrc\utils\tensor_numpy.cpp:212.)
  tensor = torch.from_numpy(np_img)
Process Process-1:
Traceback (most recent call last):
  File "C:\Users\<username>\miniconda3\envs\personalive\lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Users\<username>\miniconda3\envs\personalive\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\<username>\Program_Files\PersonaLive\webcam\vid2vid_trt.py", line 138, in generate_process
    video = pipeline.process_input(images)
  File "C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\<username>\Program_Files\PersonaLive\src\wrapper_trt.py", line 257, in process_input
    mot_bbox_param, kps_ref, kps_frame1, kps_dri = self.pose_encoder.interpolate_kps_online(self.ref_cond_tensor, tgt_cond_tensor, num_interp=12+1)        
  File "C:\Users\<username>\Program_Files\PersonaLive\src\liveportrait\motion_extractor.py", line 140, in interpolate_kps_online
    kp1 = self.detector(ref.to(self.dtype))
  File "C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\<username>\Program_Files\PersonaLive\src\liveportrait\convnextv2.py", line 120, in forward
    x = self.forward_features(x)
  File "C:\Users\<username>\Program_Files\PersonaLive\src\liveportrait\convnextv2.py", line 116, in forward_features
    x = self.stages[i](x)
  File "C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\torch\nn\modules\container.py", line 215, in forward
    input = module(input)
  File "C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\<username>\Program_Files\PersonaLive\src\liveportrait\convnextv2.py", line 41, in forward
    x = self.grn(x)
  File "C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\<username>\Program_Files\PersonaLive\src\liveportrait\util.py", line 339, in forward
    Gx = torch.norm(x, p=2, dim=(1, 2), keepdim=True)
  File "C:\Users\<username>\miniconda3\envs\personalive\lib\site-packages\torch\functional.py", line 1611, in norm
    return torch.linalg.vector_norm(input, _p, _dim, keepdim, dtype=dtype)
RuntimeError: CUDA error: invalid argument
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ..\c10\cuda\CUDAException.cpp:44 (most recent call first):
00007FFB5AC3AF4200007FFB5AC3AEE0 c10.dll!c10::Error::Error [<unknown file> @ <unknown line number>]
00007FFB5AC3AACA00007FFB5AC3AA70 c10.dll!c10::detail::torchCheckFail [<unknown file> @ <unknown line number>]
00007FFB8CF6DD0F00007FFB8CF6DA10 c10_cuda.dll!c10::cuda::c10_cuda_check_implementation [<unknown file> @ <unknown line number>]
00007FFA3A0FFC0100007FFA3A08DAF0 torch_cuda.dll!at::native::record_stream_cuda [<unknown file> @ <unknown line number>]
00007FFA3A1145C500007FFA3A08DAF0 torch_cuda.dll!at::native::record_stream_cuda [<unknown file> @ <unknown line number>]
00007FFA3AD3FB1B00007FFA3AD3E3C0 torch_cuda.dll!at::native::masked_select_out_cuda [<unknown file> @ <unknown line number>]
00007FFAAAA952E000007FFAAAA95040 torch_cpu.dll!at::native::structured_linalg_vector_norm_out::impl [<unknown file> @ <unknown line number>]
00007FFA3AC0955300007FFA3ABB3A10 torch_cuda.dll!at::cuda::where_outf [<unknown file> @ <unknown line number>]
00007FFA3AAF9F6100007FFA3AA92870 torch_cuda.dll!at::cuda::bucketize_outf [<unknown file> @ <unknown line number>]
00007FFAAB46EE3700007FFAAB46ED70 torch_cpu.dll!at::_ops::linalg_vector_norm::redispatch [<unknown file> @ <unknown line number>]
00007FFAAC9DFBA400007FFAAC6569A0 torch_cpu.dll!torch::autograd::CopySlices::~CopySlices [<unknown file> @ <unknown line number>]
00007FFAAC9BB53800007FFAAC6569A0 torch_cpu.dll!torch::autograd::CopySlices::~CopySlices [<unknown file> @ <unknown line number>]
00007FFAAB40174300007FFAAB401570 torch_cpu.dll!at::_ops::linalg_vector_norm::call [<unknown file> @ <unknown line number>]
00007FFB44ADB2D400007FFB448A5B80 torch_python.dll!THPPointer<_frame>::release [<unknown file> @ <unknown line number>]
00007FFB44AF36E800007FFB448A5B80 torch_python.dll!THPPointer<_frame>::release [<unknown file> @ <unknown line number>]
00007FFB5AF5142D00007FFB5AF506D0 python310.dll!PyCFunction_GetFlags [<unknown file> @ <unknown line number>]
00007FFB5AF0C2C800007FFB5AF0C190 python310.dll!PyObject_MakeTpCall [<unknown file> @ <unknown line number>]
00007FFB5B020E5200007FFB5B020A70 python310.dll!PyEval_GetFuncDesc [<unknown file> @ <unknown line number>]
00007FFB5B01D20E00007FFB5B01A3C0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFB5B01F52F00007FFB5B01A3C0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFB5AF0C67E00007FFB5AF0C640 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FFB5B0188E100007FFB5B018810 python310.dll!PyOS_URandomNonblock [<unknown file> @ <unknown line number>]
00007FFB5B020E5200007FFB5B020A70 python310.dll!PyEval_GetFuncDesc [<unknown file> @ <unknown line number>]
00007FFB5B01D20E00007FFB5B01A3C0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFB5B01F52F00007FFB5B01A3C0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFB5AF0C67E00007FFB5AF0C640 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FFB5AF0E96100007FFB5AF0E560 python310.dll!PyCell_Set [<unknown file> @ <unknown line number>]
00007FFB5AF0EBAA00007FFB5AF0EA50 python310.dll!PyMethod_Self [<unknown file> @ <unknown line number>]
00007FFB5B02105200007FFB5B020A70 python310.dll!PyEval_GetFuncDesc [<unknown file> @ <unknown line number>]
00007FFB5B01BFB000007FFB5B01A3C0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFB5B01F52F00007FFB5B01A3C0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFB5AF0C67E00007FFB5AF0C640 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FFB5AF0E96100007FFB5AF0E560 python310.dll!PyCell_Set [<unknown file> @ <unknown line number>]
00007FFB5AF0EBAA00007FFB5AF0EA50 python310.dll!PyMethod_Self [<unknown file> @ <unknown line number>]
00007FFB5B02105200007FFB5B020A70 python310.dll!PyEval_GetFuncDesc [<unknown file> @ <unknown line number>]
00007FFB5B01BFB000007FFB5B01A3C0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFB5B01F52F00007FFB5B01A3C0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFB5AF0C67E00007FFB5AF0C640 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FFB5AF0C0D400007FFB5AF0C000 python310.dll!PyObject_FastCallDictTstate [<unknown file> @ <unknown line number>]
00007FFB5AF0C8F200007FFB5AF0C850 python310.dll!PyObject_Call_Prepend [<unknown file> @ <unknown line number>]
00007FFB5AF7A11400007FFB5AF76670 python310.dll!PyType_Ready [<unknown file> @ <unknown line number>]
00007FFB5AF0C2C800007FFB5AF0C190 python310.dll!PyObject_MakeTpCall [<unknown file> @ <unknown line number>]
00007FFB5B020E5200007FFB5B020A70 python310.dll!PyEval_GetFuncDesc [<unknown file> @ <unknown line number>]
00007FFB5B01D19D00007FFB5B01A3C0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFB5B01F52F00007FFB5B01A3C0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFB5AF0C67E00007FFB5AF0C640 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FFB5AF0E96100007FFB5AF0E560 python310.dll!PyCell_Set [<unknown file> @ <unknown line number>]
00007FFB5AF0EBAA00007FFB5AF0EA50 python310.dll!PyMethod_Self [<unknown file> @ <unknown line number>]
00007FFB5B02105200007FFB5B020A70 python310.dll!PyEval_GetFuncDesc [<unknown file> @ <unknown line number>]
00007FFB5B01BFB000007FFB5B01A3C0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFB5B01F52F00007FFB5B01A3C0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFB5AF0C67E00007FFB5AF0C640 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FFB5AF0E96100007FFB5AF0E560 python310.dll!PyCell_Set [<unknown file> @ <unknown line number>]
00007FFB5AF0EBAA00007FFB5AF0EA50 python310.dll!PyMethod_Self [<unknown file> @ <unknown line number>]
00007FFB5B02105200007FFB5B020A70 python310.dll!PyEval_GetFuncDesc [<unknown file> @ <unknown line number>]
00007FFB5B01BFB000007FFB5B01A3C0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFB5B01F52F00007FFB5B01A3C0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFB5AF0C67E00007FFB5AF0C640 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FFB5AF0C0D400007FFB5AF0C000 python310.dll!PyObject_FastCallDictTstate [<unknown file> @ <unknown line number>]
00007FFB5AF0C8F200007FFB5AF0C850 python310.dll!PyObject_Call_Prepend [<unknown file> @ <unknown line number>]
00007FFB5AF7A11400007FFB5AF76670 python310.dll!PyType_Ready [<unknown file> @ <unknown line number>]
00007FFB5AF0C2C800007FFB5AF0C190 python310.dll!PyObject_MakeTpCall [<unknown file> @ <unknown line number>]
00007FFB5B020E5200007FFB5B020A70 python310.dll!PyEval_GetFuncDesc [<unknown file> @ <unknown line number>]
00007FFB5B01CC0D00007FFB5B01A3C0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]

[MaraScott]

I could make it worked at 2fps

with

class GRN(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.gamma = nn.Parameter(torch.zeros(1, 1, 1, dim))
        self.beta = nn.Parameter(torch.zeros(1, 1, 1, dim))

    def forward(self, x):
        # TEMP: run GRN on CPU to avoid triggering CUDA kernels
        x_cpu = x.detach().float().cpu()
        Gx = torch.sqrt(torch.sum(x_cpu * x_cpu, dim=(1, 2), keepdim=True) + 1e-6)
        Nx = Gx / (Gx.mean(dim=-1, keepdim=True) + 1e-6)
        y = (self.gamma.cpu() * (x_cpu * Nx) + self.beta.cpu() + x_cpu)

        return y.to(device=x.device, dtype=x.dtype)

[Jeremy8776]

Have you tried removing pycuda.autoinit from engine_model.py

The issue im seeing is that pycuda.autoinit creates a separate CUDA context that conflicts with PyTorch's context on Windows. After TensorRT loads, this corrupted context breaks cuBLAS/cuDNN operations in the GRN layer.

Instead of:
import pycuda.autoinit

Remove that import and change make_context() to use PyTorch's existing context:

cuda.init()
self.device = cuda.Device(self.device_int)
self.ctx = self.device.retain_primary_context()
self.ctx.push()

Should keep GRN on GPU.

[MaraScott]

@MaraScott

How are you going about CUDA tensors not being shared between processes on windows. Im stuck.

I have no clue.