[Question] SKRL - How to add new parameters in tensorboard logging? (managed RL)

[AndreaRossetto]

Question

Hi,
I need to add new log values during training, I have all the rewards but I need also to log other things, for example the joints torque but I'm not able to find how I can do it.

Where I can find an example code to do that?

[RandomOakForest]

Thank you for posting this. Have you tried track_data? SKRL allows tracking of any custom scalar metric by simply passing the name and value to track_data(). For example,

agent.track_data("Torques / mean_joint_torque", mean_joint_torque)

or

agent.writer.add_scalar("Torques / joint_torque", joint_torque_value, global_step=step)

[AndreaRossetto]

I've found it but I don't know where to put it. I've to create an CurrTerm() with that funcion or I have to modify something else? where I find the "agent" in a Managed env?

[Toni-SM]

Hi @AndreaRossetto

I don't know manager-based workflow in detail, but similar to the direct workflow, the is the self.extras["log"] dict is used to log additional/custom information to TensorBoard.

IsaacLab/source/isaaclab/isaaclab/envs/manager_based_rl_env.py

Line 367 in 3a0db9d

self.extras["log"] = dict()

Now, we need to ask to a manager-based expert about how to access it from the task config.
@Mayankm96 @kellyguo11 @ooctipus for viz

[AndreaRossetto]

@Mayankm96 @kellyguo11 @ooctipus No one can help me ?

[RandomOakForest]

Our apologies for the delay. The team has been caught up with conferences and whatnot. I'll move this post to our Discussions for follow up. Here are a few possible next steps.

When using a Manager-Based RL environment (via ManagerBasedRLEnv in Isaac Lab) with SKRL, you normally don’t create a CurrTerm() or similar; instead, you hook your custom logging into the agent that’s running. However, since in a managed training setup (like those launched from scripts/reinforcement_learning/skrl/train.py), the agent is not defined inside the environment, you must access it through the SKRL trainer script rather than modifying the environment itself.

Here’s how to integrate your custom logging like track_data() in this setup:

1. Where the agent lives

In Isaac Lab’s SKRL integration, the agent is instantiated in the training script, not inside your environment configuration.
If you open the standard SKRL training script (refer to scripts/reinforcement_learning/skrl/train.py), you’ll find something like:

from skrl.trainers.torch import SequentialTrainer
from skrl.agents.torch.ppo import PPO

agent = PPO(models=models,
             memory=memory,
             cfg=agent_cfg,
             observation_space=env.observation_space,
             action_space=env.action_space,
             device=env.device)

trainer = SequentialTrainer(env=env, agents=agent, cfg=train_cfg)
trainer.train()

This means your agent object is the PPO instance (or another algorithm) passed into the SequentialTrainer.

2. How to Add Custom Logs

You can inject your custom logging code in one of these ways:

A. Modify the Trainer Loop

If you have access to the SKRL trainer code (e.g., you cloned Isaac Lab and run your own training scripts), you can subclass the trainer and add custom hooks:

from skrl.trainers.torch import SequentialTrainer

class CustomTrainer(SequentialTrainer):
    def train(self):
        while self.progress < 1.0:
            # perform the usual SKRL training step
            super().train_step()

            # your custom log data
            torques = self.env.scene.data.applied_torque.mean().item()
            self.agents[0].track_data("Torques / mean_joint_torque", torques)

Then call:

trainer = CustomTrainer(env=env, agents=[agent], cfg=cfg)
trainer.train()

B. Inside a Reward Term (less typical)

If you only want to log something from environment internals without changing the trainer, you can instrument your reward function or observation update (inside your task’s MDP file):

def my_custom_reward(env, **kwargs):
    torques = env.scene.data.applied_torque.mean().item()
    env.agent.track_data("Torques / mean_joint_torque", torques)
    return 0.0

However, this approach only works if your environment has access to the agent (which a ManagerBasedRLEnv typically doesn’t by default). For reliability, prefer modifying or subclassing the trainer.

3. Cleaner Option: Use Callbacks

SKRL supports experiment callbacks you can define in your agent configuration (cfg).
You can define one like this:

def log_joint_torques(agent, timestep, timesteps):
    torque_mean = agent.env.scene.data.applied_torque.mean().item()
    agent.track_data("Torques / mean_joint_torque", torque_mean)

agent_cfg["experiment"]["callbacks"] = {"on_step_end": [log_joint_torques]}

This will automatically call your log_joint_torques function every SKRL step, giving access to both the environment and the agent contexts.

---

Summary

Method	Recommended For	Access to agent	Code location
track_data() in trainer loop	Most flexible	Yes	Modify SKRL trainer or subclass
track_data() in reward term	Quick experiments	Sometimes	Inside mdp.py or RewTermCfg
Callback config (agent_cfg["experiment"]["callbacks"])	Clean and modular	Yes	In agent_cfg before trainer creation

This setup is compatible with Isaac Lab version ≥2025.09 and SKRL 1.4.3, based on NVIDIA’s official documentation for manager-based RL environments and SKRL integration within Isaac Lab.¹²

Footnotes

1. https://isaac-sim.github.io/IsaacLab/main/source/tutorials/03_envs/create_manager_rl_env.html ↩

2. https://isaac-sim.github.io/IsaacLab/main/source/overview/reinforcement-learning/rl_existing_scripts.html ↩

[AndreaRossetto]

@RandomOakForest The last way it seems to be the best, so for example if I wanna add it I have to do something like this?

def log_joint_torques(agent, timestep, timesteps):
    torque_mean = agent.env.scene.data.applied_torque.mean().item()
    agent.track_data("Torques / mean_joint_torque", torque_mean)

@hydra_task_config(args_cli.task, agent_cfg_entry_point)
def main(env_cfg: ManagerBasedRLEnvCfg | DirectRLEnvCfg | DirectMARLEnvCfg, agent_cfg: dict):
    """Train with skrl agent."""
    # override configurations with non-hydra CLI arguments
    env_cfg.scene.num_envs = args_cli.num_envs if args_cli.num_envs is not None else env_cfg.scene.num_envs
    env_cfg.sim.device = args_cli.device if args_cli.device is not None else env_cfg.sim.device

    # multi-gpu training config
    if args_cli.distributed:
        env_cfg.sim.device = f"cuda:{app_launcher.local_rank}"
    # max iterations for training
    if args_cli.max_iterations:
        agent_cfg["trainer"]["timesteps"] = args_cli.max_iterations * agent_cfg["agent"]["rollouts"]
    agent_cfg["trainer"]["close_environment_at_exit"] = False
    # configure the ML framework into the global skrl variable
    if args_cli.ml_framework.startswith("jax"):
        skrl.config.jax.backend = "jax" if args_cli.ml_framework == "jax" else "numpy"

    # randomly sample a seed if seed = -1
    if args_cli.seed == -1:
        args_cli.seed = random.randint(0, 10000)

    # set the agent and environment seed from command line
    # note: certain randomization occur in the environment initialization so we set the seed here
    agent_cfg["seed"] = args_cli.seed if args_cli.seed is not None else agent_cfg["seed"]
    env_cfg.seed = agent_cfg["seed"]

    # specify directory for logging experiments
    log_root_path = os.path.join("logs", "skrl", agent_cfg["agent"]["experiment"]["directory"])
    log_root_path = os.path.abspath(log_root_path)
    print(f"[INFO] Logging experiment in directory: {log_root_path}")
    # specify directory for logging runs: {time-stamp}_{run_name}
    log_dir = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") + f"_{algorithm}_{args_cli.ml_framework}"
    # The Ray Tune workflow extracts experiment name using the logging line below, hence, do not change it (see PR #2346, comment-2819298849)
    print(f"Exact experiment name requested from command line: {log_dir}")
    if agent_cfg["agent"]["experiment"]["experiment_name"]:
        log_dir += f'_{agent_cfg["agent"]["experiment"]["experiment_name"]}'
    # set directory into agent config
    agent_cfg["agent"]["experiment"]["directory"] = log_root_path
    agent_cfg["agent"]["experiment"]["experiment_name"] = log_dir

I add here the code to log the new parameters

agent_cfg["agent"]["experiment"]["callbacks"] = {"on_step_end": [log_joint_torques]}

    # update log_dir
    log_dir = os.path.join(log_root_path, log_dir)

    # dump the configuration into log-directory
    dump_yaml(os.path.join(log_dir, "params", "env.yaml"), env_cfg)
    dump_yaml(os.path.join(log_dir, "params", "agent.yaml"), agent_cfg)
    dump_pickle(os.path.join(log_dir, "params", "env.pkl"), env_cfg)
    dump_pickle(os.path.join(log_dir, "params", "agent.pkl"), agent_cfg)

    # get checkpoint path (to resume training)
    resume_path = retrieve_file_path(args_cli.checkpoint) if args_cli.checkpoint else None

    # create isaac environment
    env = gym.make(args_cli.task, cfg=env_cfg, render_mode="rgb_array" if args_cli.video else None)

    # convert to single-agent instance if required by the RL algorithm
    if isinstance(env.unwrapped, DirectMARLEnv) and algorithm in ["ppo"]:
        env = multi_agent_to_single_agent(env)

    # wrap for video recording
    if args_cli.video:
        video_kwargs = {
            "video_folder": os.path.join(log_dir, "videos", "train"),
            "step_trigger": lambda step: step % args_cli.video_interval == 0,
            "video_length": args_cli.video_length,
            "disable_logger": True,
        }
        print("[INFO] Recording videos during training.")
        print_dict(video_kwargs, nesting=4)
        env = gym.wrappers.RecordVideo(env, **video_kwargs)

    # wrap around environment for skrl
    env = SkrlVecEnvWrapper(env, ml_framework=args_cli.ml_framework)  # same as: `wrap_env(env, wrapper="auto")`

    # configure and instantiate the skrl runner
    # https://skrl.readthedocs.io/en/latest/api/utils/runner.html
    runner = Runner(env, agent_cfg)

    # load checkpoint (if specified)
    if resume_path:
        print(f"[INFO] Loading model checkpoint from: {resume_path}")
        runner.agent.load(resume_path)

    # run training
    runner.run() # Default mode is train

    # close the simulator
    env.close()

Is that correct?

[AndreaRossetto]

I added the line agent_cfg["agent"]["experiment"]["callbacks"] = {"on_step_end": [log_joint_torques]} in my code but I cannot see the log in tensorboard

[AndreaRossetto]

I'm still having problems making the parameters appear in tensorboard