[Question] Unit testing of IsaacLab scripts

[XiRouz]

Question

I couldn't find any info in IsaacLab documentation related to unit testing of standalone scripts. Are there any recommendations regarding this topic?

[garylvov]

I think that the scripts are difficult to test as they don't explicitly return anything; there's just one main function. So, to the best of my knowledge, there currently aren't any tests.

Any tests for the current scripts would have to check if logs were correctly created where expected.

This makes the current Ray tune integration a little bit messy as the training stats must be scraped from the logs. While I was working on the Ray tune integration, I ran into the issue of nothing being returned explicitly from the scripts, as well as the fact that the script directory isn't really a Python module (no __init__.py).

It would be nice for the scripts to be a python module, with ways to introspect them, like getting training stats for tests or the Ray integration, or being able to stop start/stop envs programmatically without using bash. This could make tests happen for the scripts, as well as potentially making the Ray integration a bit cleaner.

However, I suspect this change would be a significant lift for the dev team. Currently, the scripts do a good job at delegating to their wrappers (like rsl rl, rl games, etc). Changing the way these scripts operate could require changing the individual wrapper libraries runners, or could require custom changes to allow returning some object to interact with in a test beyond the logs.

I would make the case that this change isn't worth the trouble when there are other items with potentially higher priority and the scripts are mostly reliable. However, I can see the other side here. I would suggest creating a proposal about changing the scripts (or making special tests that read output log directories that are already created) if you feel passionately about adding tests to the scripts ;)

[garylvov]

Also see related comment here: #2346 (comment)

[XiRouz]

Hey @garylvov, thanks for the reply!

I am sorry for how abstract my question sounds, I needed to be more precise about what do I want to test. I don't use IsaacLab for RL tasks, it is required for me to make a reusable script with simulation environment configuration for tasks not related to ML or RL.

What about testing general stuff like initialization of the environment (i.e. are all the camera components initialized properly), ROS communication, events and observations, etc?

I've seen a PR mentioning swapping from unittest lib to pytest lib for internal IsaacLab tests, I guess pytest is the right library to go with for unit testing then?

[garylvov]

For general things, we have tests!

For example, for core stuff, tests are housed at https://github.com/isaac-sim/IsaacLab/tree/main/source/isaaclab/test

All tests can be ran locally with

 ./isaaclab.sh --test

Also feel free to check out the testing CI at
https://github.com/isaac-sim/IsaacLab/blob/main/.github/actions/run-tests/action.yml