[DannyYuyang-quic]

Summary

This RFC integrates the Lerobot-ExecuTorch simulation framework.
The LeRobot SmolVLA policy is a example to run fully delegated on the Qualcomm HTP through ExecuTorch QNN backend and demonstrates end-to-end closed-loop robot control in the LIBERO benchmark.

Design

LeRobot and ExecuTorch run in separate Python environments and form a closed loop over TCP. Terminal B owns the LIBERO benchmark and sends an observation whenever its action queue is empty. Terminal A runs SmolVLA and returns a 50-action chunk. Terminal B executes the chunk one action per env.step(), then sends the next observation:

Co-authors: @haowhsu-quic † @paul1106 †
† Equal contribution.

Quick Start

Environment Setup

Terminal A (ExecuTorch environment)
Prepare an ExecuTorch environment with the Qualcomm QNN backend successfully built

Terminal B (LeRobot environment):

conda activate VLA
pip install lerobot
pip install libero
pip install num2words

On a headless Linux host, robosuite renders through EGL. If the system libEGL is missing, eval.py fails with 'NoneType' object has no attribute 'eglQueryString'. To fix it:

sudo apt-get install -y libegl1 libegl-mesa0 libgl1-mesa-dri mesa-utils

Run

Open two terminals and start both processes. Terminal A hosts the ExecuTorch + QNN inference service, while Terminal B runs the LeRobot LIBERO rollout client. The two processes communicate over TCP to form a closed-loop evaluation pipeline.
Start Terminal A first, then launch Terminal B while Terminal A remains running.

Terminal A (ExecuTorch env):
A single run executes the full pipeline: export → FP32 rollout → PTQ calibration → QDQ rollout → lowering → HTP rollout.

python examples/qualcomm/robotics/scripts/smolvla/e2e.py \
    --device ${SERIAL_NUM} \
    --soc-model SM8750 \
    --artifact ./smolvla_qnn \
    --build-folder build-android

Tip

Add --pre_gen_pte ./smolvla_qnn to skip export and compilation and run only the HTP rollout.

Terminal B (LeRobot env):

python examples/qualcomm/robotics/scripts/smolvla/eval.py \
    --rollout-benchmark libero_10 \
    --rollout-task-id 9 \
    --rollout-episode-seed 0 \
    --rollout-max-steps 500 \
    --rollout-video-fps 30 \
    --connect-timeout-seconds 7200 \
    --rollout-output-directory outputs

--connect-timeout-seconds must cover the time Terminal A spends exporting, calibrating and compiling between rollouts.

Results

LIBERO-10, task 9: "put the yellow and white mug in the microwave and close it", episode seed 0.

Runs on	Task Success	Steps to complete task	Plans	Mean plan (ms/plan)	Model Execution Cost (s/task)
Eager mode (exported graph)	✅	255	6	1227.89	~7.3
HTP / SM8750	✅	259	6	703.28	~4.2
XNNPACK	✅	255	6	4120.56	~24.72

Notes:

• QDQ simulates quantization with fake-quant ops on the CPU, so it validates accuracy, not speed.
• HTP takes 4 more steps than FP32 and QDQ, but all three modes complete the task.
• For XNNPACK, this is not the most optimized version.

Rollout Videos

libero_10_task_09_seed_000_htp_module_rollout.mp4 HTP / SM8750

libero_10_task_09_seed_000_qdq_module_rollout.mp4 QDQ Module

libero_10_task_09_seed_000_exported_module_rollout.mp4 Eager Mode

Quantization Precision for SmolVLA

Module	Precision
Vision Encoder	FP16
Prefix Model	W8A16, per-channel weights
Action Expert	W8A16, per-channel weights

cc: @psiddh @JacobSzwejbka @shoumikhin @cccclai @shewu-quic @haowhsu-quic @paul1106 @winskuo-quic @harshs-qti

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24829

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⚠️ 25 Awaiting Approval

As of commit 250cc66 with merge base f2575be ():

AWAITING APPROVAL - The following workflows need approval before CI can run:

• Build Aarch64 Linux CUDA Wheels (gh)
• Build Aarch64 Linux Native Wheels (gh)
• Build Aarch64 Linux Wheels (gh)
• Build documentation (gh)
• Build Linux CUDA Wheels (gh)
• Build Linux Native Wheels (gh)
• Build Linux Wheels (gh)
• Build macOS Wheels (gh)
• Build Presets (gh)
• Build Windows CUDA Wheels (gh)
• Build Windows Wheels (gh)
• Cadence Build & Test (gh)
• Lint (gh)
• periodic (gh)
• pull (gh)
• Test ARM Backend (gh)
• Test CoreML Backend (gh)
• Test Cortex-M Backend (gh)
• Test CPU Backend (gh)
• Test NXP Backend (gh)
• Test QNN Backend (gh)
• Test QNN Windows MSVC build (gh)
• Test Vulkan Backend (gh)
• Test WebGPU Backend (gh)
• Test XNNPACK Backend (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[shoumikhin]

Thanks @DannyYuyang-quic, @haowhsu-quic and @paul1106. It is great to see SmolVLA finish a LIBERO task on the HTP of a real device, quantized, with videos to check it.

Some questions about the overall design first, because they affect most of the files:

1. Model code. The PR rebuilds SmolVLA on top of static_llama and loads LeRobot's weights. That follows your LLM examples, but every LeRobot policy would need its own rebuild, and each rebuild has to follow LeRobot's changes. In Qualcomm (QNN): LeRobot policies that do not run yet, and what blocks them #24668 we export LeRobot's own SmolVLAPolicy instead. It lowers to QNN and matches eager PyTorch on the x86 emulator with Qualcomm AI Engine Direct - Fix float16 RMSNorm, float64 and int64 data, and constant outputs #24428, Qualcomm AI Engine Direct - Keep positional sin and cos in float32 #24601 (both still open) and Qualcomm AI Engine Direct - Decompose kept-whole ops that HTP leaves on CPU #24429. That path is fp16 only and has not run on a device, so your W8A16 recipe and device run go further. What blocked LeRobot's own code for you? We would like to fix it, in LeRobot or in ExecuTorch.

2. Two environments. LeRobot already ships lerobot-env-server, which serves LIBERO from its own environment, and lerobot-policy-server for remote inference. Could the LIBERO run use those instead of a new TCP server and client? If something is missing, adding it in LeRobot would help every backend.

3. Scope and chips. About 4,400 lines is a lot to review at once. Once the design settles, could you split it? For example: model export and the quantization recipe first, then the runner, then the LIBERO loop. Also, do you plan to try a Dragonwing robotics board, such as the IQ-9075? Does the existing QCS9100 entry cover it?

Happy to help on the LeRobot side.

[shoumikhin]

@claude review effort=max

[claude]

Claude finished @shoumikhin's task in 55m 34s —— View job

---

PR Review: #24829

Request Changes: the SmolVLA/LIBERO QNN flow hangs whenever either side exits (FramedSocket spins on EOF, accept() never times out, and no SHUTDOWN is ever sent), --compile_only still needs the simulator, and none of the ~4.5k new lines are tested.

Detailed review

Summary

This PR adds examples/qualcomm/robotics/. It rebuilds lerobot/smolvla_libero from static_llama building blocks as three QNN graphs (an FP16 vision encoder plus a W8A16 prefix and action expert), runs them with a new qnn_smolvla_runner, and drives closed-loop LIBERO rollouts from a separate LeRobot process over a new TCP/msgpack link.

Problems found:

• The transport hangs when either side exits.
• A normal run ends with Terminal B erroring out.
• Several CLI flags don't do what they say.
• Nothing is tested.

The first two design questions in the thread (exporting LeRobot's policy, and reusing LeRobot's servers) correspond to two of the design findings below.

File names without a directory refer to files this PR adds under examples/qualcomm/robotics/.

Code Quality

• [must-fix] _recv_exact never detects EOF (frame_socket.py:178-182).

  • Once the peer closes, recv() returns b"" immediately. The loop then spins at 100% CPU and chunks grows without bound. The read timeout never fires.
  • Both sides hit this:
    • If Terminal B exits mid-rollout, the except ConnectionError handlers at e2e.py:146/289/456 never run.
    • If Terminal A raises (e.g. parse_runner_timing after a failed device run), Terminal B spins inside client.exchange.
  • Fix: raise ConnectionError on an empty chunk. Fix this →

• [major] Session lifecycle (e2e.py:129-133, 199-204, 272-276, 438-444, eval.py:480-562, server.py:142-199):

  • Terminal A never sends WorkerType.SHUTDOWN, although eval.py:3-24 documents it. After the last phase, Terminal B keeps reconnecting until --connect-timeout-seconds runs out (7200 s in the documented command), then prints [worker] Process failed and exits non-zero.
  • Each phase opens its own listener, and nothing listens between phases:
    • compile() plus adb.push() before the HTP rollout can take longer than Terminal B's default 600 s connect timeout (eval.py:138).
    • Terminal B then exits, and Terminal A blocks forever in listener.accept() (server.py:199, no timeout).
  • When an except ConnectionError: break handler does fire, Terminal A treats the truncated rollout as complete and moves into the next phase's accept(), where it blocks.

  Fix: use one connection for the whole session, end it with SHUTDOWN, and fail fast when the peer disappears. Fix this →

• [major] Design: the static_llama subclasses mostly bypass their parent (prefix_decoder.py:112-152, action_expert.py:112-157, config.py:141, examples/qualcomm/oss_scripts/llama/model/static_llama.py:670-679).

  • The parent constructor builds layers, tok_embeddings and output (sized by the VLM vocabulary, config.py:141), which are then replaced or never used.
  • The masked softmax and MASKED_SCORE_MARGIN are duplicated.
  • Only apply_rope_emb, attn_softmax, feed_forward, the norms and the RoPE tables are actually reused.
  • This is the substance of the first question in the thread (why not export LeRobot's SmolVLAPolicy, cf. Qualcomm (QNN): LeRobot policies that do not run yet, and what blocks them #24668). If the rebuild stays, compose those pieces into standalone modules instead of subclassing.

• [major] Design: a ~1,000-line custom transport with no written wire schema (transport/, remote_inference/).

  • examples/qualcomm scripts already talk to their callers through multiprocessing.connection (54 scripts plus backends/qualcomm/tests/test_qnn_delegate.py).
  • Listener/Client with send_bytes/recv_bytes(maxlength) give framing, EOF detection and a size cap, without pickle.
  • This is the second question in the thread (why not lerobot-env-server/lerobot-policy-server).
  • If a custom link stays, document the message sequence in worker.py: worker selection, per-plan payloads, WORKER_COMPLETE, and the acks.

• The serve loop is copy-pasted three times (e2e.py:129-171, 272-311, 438-510).

  • The HTP copy builds SmolVLAPolicy(None, ...) (e2e.py:425), so calling predict_action_chunk (policy.py:128-131) on it raises a TypeError.
  • Fix: wrap the device runner in a model callable and share one serve_rollout().

• The calibration early exit deadlocks (e2e.py:216-231, 242).

  • On WORKER_COMPLETE, the loop breaks without send(True) and then calls recv() again. Both sides then wait out their 3000 s read timeouts.
  • The branch is unreachable today. Remove it, or ack and skip the second recv().

• (low confidence) The whole chunk is executed. rollout.py:129 queues all chunk_size actions and never reads n_action_steps, which may diverge from LeRobot's select_action. Please confirm against the policy config.

• The LIBERO env is never closed (eval.py:386-399, rollout.py). A full run creates three envs in one process (for the exported, QDQ and HTP rollouts) and never calls close().

• Calibration data (eval.py:315-318, libero.yaml):

  • LeRobotDataset("lerobot/libero") is built without episodes=, so the whole dataset downloads while Terminal A waits in recv().
  • max_episodes: 5 at observation_interval: 20 then keeps only ~75-125 frames of one task. That is thin coverage for W8A16 PTQ.
  • Is task_id: 9 of the merged dataset the same task as libero_10 task 9? If so, the only evaluated task is also the calibration task.
  • seed is unused.

• config.py:

  • 38-47 hard-codes num_cameras, image_tokens_per_camera and prefix_layers without checking them against the checkpoint. prefix_layers=16 stays fixed even though num_vlm_layers is read at 178.
  • 165-166 (and action_expert.py:171) compute % self_attn_every_n_layers, which raises ZeroDivisionError when it is 0.
  • 184 and 186 call getattr on text_config, which CLAUDE.md disallows. LeRobot's get_intermediate_size uses the constants 4 and 256.
  • vlm_config (72-74) re-runs AutoConfig.from_pretrained on every access.

Infrastructure

• BuildQuantIo() is a no-op (e2e.py:376-382, backends/qualcomm/_passes/build_quant_io.py:38-41).
  • The pass only rewrites IO tagged QCOM_QUANTIZED_IO (via tag_quant_io). Nothing is tagged, so every PTE boundary stays float.
  • As a result, the 32 prefix K/V tensors are dequantized at the prefix output and requantized by the action expert on each of the 10 Euler steps.
  • Fix: tag the K/V IO as 16-bit with identical encodings on both sides, or drop the pass. The runner's same_contract compares dtype, size and dim order, but not scale or offset.
• Terminal B's documented install doesn't cover eval.py's imports.
  • eval.py:56-77 imports executorch.examples..., which pip install lerobot libero num2words doesn't provide.
  • codec.py imports msgpack, which no requirements file declares. Terminal A needs it too.

Testing

• No test exercises any of the 4,453 added lines. At minimum, add:
  • a transport test over socket.socketpair() covering framing, EOF and oversized frames;
  • a parity test of SmolVLAPolicy.predict_action_chunk against LeRobot's, with fixed noise;
  • a compile-only entry in test_qnn_delegate.py, which is blocked on the --compile_only item under API Design.
• The results table rests on one seed of one task, and its XNNPACK row has no code path in this PR.

API Design

• [major] --compile_only still needs Terminal B and LIBERO (e2e.py:115, 576).

  • main() always calls quantize(). Its exported and QDQ rollouts (skip_rollout = False is hard-coded) and its calibration all wait on Terminal B.
  • Fix: make the rollouts opt-in and let calibration read cached observations, so the PTEs can be built without a simulator, including in CI.

• e2e.py CLI:

  • --server-host/--server-port (533-534) are parsed and then ignored in favour of SERVER_HOST/SERVER_PORT (61-62).
  • --device-output-dir (539) is unused, and --verbose (530) only feeds the conflict check at 547-548.
  • The model id is hard-coded (553), while eval.py takes --model-id.
  • Flag spelling mixes styles, e.g. --soc-model and --compile_only.
  • --soc-model, --build-folder, --adb-host and the rest are re-declared instead of coming from setup_common_args_and_variables (backends/qualcomm/export_utils.py:787, used by 56 scripts).
  • --soc-model defaults to SM8850, while the PR was validated on SM8750.

  Fix this →

• The runner's --num_steps isn't tied to the PTE (qnn_smolvla_runner.cpp:53, 339-345, action_expert.py:240-258).

  • The Euler time embeddings are baked into the action-expert PTE for config.num_steps. The runner instead takes the step count from a flag and only checks that it is positive (369).
  • A larger value reads past the table. A smaller one changes dt but not the baked-in times.
  • This contradicts the header comment at line 15 ("Every shape and byte size comes from the PTE MethodMeta").
  • Fix: export num_steps through constant_methods, as the llama export does for its metadata, and read it in the runner. Fix this →

• Part of the network runs on the host (policy.py:134-184, prefix_decoder.py:179-216).

  • SmolVLAPrefixEmbedding (the language-token embedding and the state projection) runs inside prepare_inputs on the host. The runner consumes its output as prefix_tail.raw, so a device can't produce a plan without a Python host.
  • This conflicts with the "fully delegated" description and matters for the IQ-9075/Dragonwing question in the thread.
  • For comparison, the llama flow lowers tok_embedding as its own PTE (examples/qualcomm/oss_scripts/llama/wrappers/llm_wrappers.py:1158-1204).

Security

• (low) Unauthenticated listener that trusts the length prefix (e2e.py:533, frame_socket.py:162-172).
  • Today the listener is loopback-only (SERVER_HOST).
  • Once --server-host is wired up with its 0.0.0.0 default, Terminal A accepts any peer that can reach the port.
  • recv() trusts the 4-byte length, so a single header can make it try to allocate up to 4 GiB.
  • Fix: default to 127.0.0.1 and cap the frame size. multiprocessing.connection also offers authkey.

Performance

• One adb round trip and a cold runner per plan (e2e.py:468-484, qnn_smolvla_runner.cpp:272-277).
  • Every plan pushes the inputs, starts a new runner process that loads all three PTEs, and pulls the output.
  • The runner's plan_ms excludes the load, while mean_plan_e2e_ms includes adb and the process start. Which one is the 703.28 ms in the results table?
  • A persistent runner that loads the PTEs once and loops over plans would remove most of that overhead.

Specific Comments

• frame_socket.py:1-20 - the module docstring is a list of scratch notes; logger (30) is unused.
• codec.py:
  • 1-10 - the docstring is garbled.
  • 17 - logger is unused.
  • 21 - the class docstring says "Bounded", but nothing is bounded.
  • 23-24 - __init__ is empty.
  • 54/78 - __nd__ is checked twice.
• server.py:244-288 - prints on every send and recv. connected/client_address (125-140) and exchange (290-299) are unused.
• client.py:66-68 - prints "Connecting to remote server" after the connection is already established; logger (9) is unused.
• worker.py:62 - the example names (exported, qdq) don't match the names e2e.py passes (exported_module_rollout, ...).
• e2e.py:
  • 101, 108 - the set_recipe calls are commented out, so the two quant_recipe.py classes are dead (and identical to each other). The two make_quantizer calls are also identical.
  • 111 - logs "Exported model" before export() runs.
  • 197-198, 259, 334 - typos: "seperate enviroment", "recievd", "perferom", "Simultae", "quatnized".
  • 263 - the QDQ rollout reuses the FP32 vision module, while the PTE runs it in FP16.
  • 315 - compile shadows the builtin.
  • 342 - compile_specs is the FP16 spec; name it that.
  • 348-353 - the None checks can never fire.
• eval.py:
  • 35 - MUJOCO_GL=glfw needs a display, whereas headless runs normally use egl.
  • 138, 177 - the defaults (600 s, 300 steps) don't match the documented run (7200 s, 500 steps).
  • 479 - typo ("works call").
• qnn_smolvla_runner.cpp:
  • 83-87, 368-374 - this is the only examples/qualcomm runner that uses exceptions; the others use ET_CHECK_MSG/ET_LOG.
  • 198-199 - prints 0x followed by a decimal number.
• htp_io.py:48-54 - push_inputs re-implements SimpleADB's adb invocation.
• rollout.py:121-122 - frames are captured before each step, so the final frame is missing from the video.
• vision_encoder.py:
  • 49 - construct CustomSmolVLMVisionEmbeddings here instead of swapping it in at 77-83.
  • 115-117 - the reshape is an identity.
• libero.py:
  • 1-19 - unused copies of eval.py's maps, placed above the imports.
  • 27-28 - stale comments.
• vla_flow_matching_exportable_module.py:
  • 29 - strict export handles lists of tensors, so that isn't the reason for strict=False.
  • 22-23 - forward is unused.
• 13 of the new Python files lack the license header, and LICENSELINT won't flag them.
• There is no README. The two-terminal setup is documented only in the PR description.

Recommendation

Request Changes

Blocking:

• the _recv_exact spin;
• the session lifecycle (no SHUTDOWN, no listener between phases, accept() without a timeout);
• --compile_only still needing the simulator;
• the missing transport and parity tests.

Before polishing the rest, please settle the two design questions in the thread: exporting LeRobot's policy vs. a static_llama rebuild, and existing LeRobot servers vs. a new transport. Also consider splitting the PR into (1) model and export, (2) runner, and (3) simulation and transport.

[DannyYuyang-quic]

Thanks @DannyYuyang-quic, @haowhsu-quic and @paul1106. It is great to see SmolVLA finish a LIBERO task on the HTP of a real device, quantized, with videos to check it.

Some questions about the overall design first, because they affect most of the files:

1. Model code. The PR rebuilds SmolVLA on top of static_llama and loads LeRobot's weights. That follows your LLM examples, but every LeRobot policy would need its own rebuild, and each rebuild has to follow LeRobot's changes. In Qualcomm (QNN): LeRobot policies that do not run yet, and what blocks them #24668 we export LeRobot's own SmolVLAPolicy instead. It lowers to QNN and matches eager PyTorch on the x86 emulator with Qualcomm AI Engine Direct - Fix float16 RMSNorm, float64 and int64 data, and constant outputs #24428, Qualcomm AI Engine Direct - Keep positional sin and cos in float32 #24601 (both still open) and Qualcomm AI Engine Direct - Decompose kept-whole ops that HTP leaves on CPU #24429. That path is fp16 only and has not run on a device, so your W8A16 recipe and device run go further. What blocked LeRobot's own code for you? We would like to fix it, in LeRobot or in ExecuTorch.

Thanks for sharing the work in progress on these issues.
We modified the policy mainly to optimize how the quantized model actually behaves on HTP. Two examples:

• Precomputed Euler time embeddings. The time embedding of every step in the fixed schedule is precomputed and selected by step index. This removes the sin/cos computation from the action expert. The expert runs the number of denoising steps per plan, so that saves the time-embedding computation in each of those runs. and with it the quantization error those ops introduce.
• The masked softmax we also use in static_llama. Masking with finfo(float32).min stretches the quantized range of the attention scores, so the valid scores collapse into the same bin. The masked softmax replaces masked scores with a finite margin below the row minimum, which keeps the range tight and solves this accuracy problem.
  In our experiments so far, LIBERO rollouts on HTP without these optimizations did not succeed, even at the higher precision we use now.

On the blocking issues: when we first enabled the original SmolVLA policy directly, we ran into many of the issues you are fixing. We can share a list of the blocking issues we hit while enabling it later.

2. Two environments. LeRobot already ships lerobot-env-server, which serves LIBERO from its own environment, and lerobot-policy-server for remote inference. Could the LIBERO run use those instead of a new TCP server and client? If something is missing, adding it in LeRobot would help every backend.

Thanks for the suggestion. Let us share some context from our side. Our TCP server-client has two goals.

• it gives us a simple way to reproduce rollouts of the exported graph, the QDQ fake-quant graph, and the .pte running in the HTP C++ runtime, all through one shared architecture. The LIBERO side stays the same and only the model behind it changes.

• ExecuTorch currently seems conflicts with the LeRobot/LIBERO environment, so we run the two sides in separate, isolated environments.

As for lerobot-policy-server, seems like it loads a PreTrainedPolicy (an nn.Module) from a checkpoint through from_pretrained, it has no native path for graph-format policies, such as an exported or fake-quant graph, or a .pte run by an on-device runtime. If it already supports any of these formats, or there is a recommended way to serve them, please let us know and we'd be happy to try it.

3. Scope and chips. About 4,400 lines is a lot to review at once. Once the design settles, could you split it? For example: model export and the quantization recipe first, then the runner, then the LIBERO loop.

No problem. Splitting the work into several smaller PRs is also part of our plan once the design settles.

4. Also, do you plan to try a Dragonwing robotics board, such as the IQ-9075? Does the existing QCS9100 entry cover it?

Good question. Which SoCs to support for robotics is still under discussion on our side. We'll follow up on this topic once a decision is made.

[shoumikhin]

Thanks @DannyYuyang-quic, this helps a lot.

Both model changes make sense for quantized HTP. Could they live as passes on the exported graph instead of a rebuilt model? The time embeddings could become a lookup table over the fixed schedule, with the step index as the input. The masked softmax could be a pattern replacement like the one in static_llama. Since the finfo.min range problem affects any quantized backend, it might even work as a generic pass. Then an export of LeRobot's own SmolVLAPolicy, and of other policies with the same patterns, would get the same accuracy, with no model copy to keep in sync.

Please share the list of blockers you hit with the original policy, ideally on #24668. We will fix them in ExecuTorch or in LeRobot.

On the environments, what exactly conflicts? If it is the torch pin, #24820 proposes a runtime-only package with no torch requirement, so the runtime can live in LeRobot's environment. You are right that lerobot-policy-server only loads PyTorch policies today. We would like to add an ExecuTorch path to LeRobot's rollout, so a QNN runner can plug in there and the LIBERO loop does not need its own transport. Until then, keeping your harness in the example is fine with me.

Thanks for planning the split. Starting with the model, export and quantization recipe sounds good.

[DannyYuyang-quic]

@shoumikhin Thanks for the feedback!

Sure, we'll make these into passes instead of rebuilding the model.

#24668 is really helpful. We'll double-check whether it covers all of the cases and share any blockers that are still not covered.

Yes, the conflicts is the torch pin, thanks for purpose torch-free runtime package.

Having an ExecuTorch path in LeRobot's rollout would let the QNN runner plug in cleanly. We'll keep the harness in the example until the ExecuTorch path in LeRobot's rollout is ready. Quick question: are there plans to also support the .pt2 format in LeRobot's rollout? It would make it much easier to compare QDQ graph module (fake-quant module) results against the on-device runner in the same LIBERO loop.

[shoumikhin]

Thanks @DannyYuyang-quic, sounds good.

On .pt2: not in the first version. It runs only .pte programs, so the rollout works without PyTorch. But the plan is to load programs through a small runtime interface, so a runtime that loads a .pt2 with torch.export.load can plug into the same loop. It needs PyTorch in that environment, which is fine for an accuracy check. Then you can run the fake-quant graph and the device runner in the same LIBERO loop, with the same seeds, and compare them.

I can add it once the ExecuTorch path is in. Until then, your harness covers it.