Skip to content

Commit

Permalink
Updating MuJoCo docstrings (#2616)
Browse files Browse the repository at this point in the history 
* Changed Ant docstring

* Changed half cheetah docstring

* Changed Hopper docstring, modified formulations for Ant, HalfCheetah

* Changed humanoid docstring

* Changed HumanoidStandup, InvertedDoublependulum, InvertedPendulum docstrings

* Updated remaining docstrings, updated names of reward terms, added notes about info
  • Loading branch information
@Markus28
Markus28 authored Feb 17, 2022
1 parent 2c0358b commit 95063a0
Show file tree
Hide file tree
Showing 9 changed files with 370 additions and 395 deletions.
137 changes: 75 additions & 62 deletions gym/envs/mujoco/ant_v3.py
View file

Large diffs are not rendered by default.

81 changes: 32 additions & 49 deletions gym/envs/mujoco/half_cheetah_v3.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,7 @@ class HalfCheetahEnv(mujoco_env.MujocoEnv, utils.EzPickle):
(connecting to the thighs) and feet (connecting to the shins).
### Action Space
The agent take a 6-element vector for actions.
The action space is a continuous `(action, action, action, action, action, action)` all in `[-1.0, 1.0]`, where `action` represents the numerical torques applied between *links*
The action space is a `Box(-1, 1, (6,), float32)`. An action represents the torques applied between *links*.
| Num | Action | Control Min | Control Max | Name (in corresponding XML file) | Joint | Unit |
|-------|--------------------------------------|---------------|----------------|---------------------------------------|-------|------|
Expand All @@ -39,43 +38,22 @@ class HalfCheetahEnv(mujoco_env.MujocoEnv, utils.EzPickle):
### Observation Space
The state space consists of positional values of different body parts of the
Observations consist of positional values of different body parts of the
cheetah, followed by the velocities of those individual parts (their derivatives) with all the positions ordered before all the velocities.
The observation is a `ndarray` with shape `(17,)` where the elements correspond to the following:
By default, observations do not include the x-coordinate of the cheetah's center of mass. It may
be included by passing `exclude_current_positions_from_observation=False` during construction.
In that case, the observation space will have 18 dimensions where the first dimension
represents the x-coordinate of the cheetah's center of mass.
Regardless of whether `exclude_current_positions_from_observation` was set to true or false, the x-coordinate
will be returned in `info` with key `"x_position"`.
However, by default, the observation is a `ndarray` with shape `(17,)` where the elements correspond to the following:
| Num | Observation | Min | Max | Name (in corresponding XML file) | Joint| Unit |
|-----|-----------------------|----------------------|--------------------|----------------------|--------------------|--------------------|
| 0 | x-coordinate of the center of mass | -Inf | Inf | rootx | slide | position (m) |
| 1 | y-coordinate of the center of mass | -Inf | Inf | rootz | slide | position (m) |
| 2 | angle of the front tip | -Inf | Inf | rooty | hinge | angle (rad) |
| 3 | angle of the back thigh rotor | -Inf | Inf | bthigh | hinge | angle (rad) |
| 4 | angle of the back shin rotor | -Inf | Inf | bshin | hinge | angle (rad) |
| 5 | angle of the back foot rotor | -Inf | Inf | bfoot | hinge | angle (rad) |
| 6 | velocity of the tip along the y-axis | -Inf | Inf | fthigh | hinge | angle (rad) |
| 7 | angular velocity of front tip | -Inf | Inf | fshin | hinge | angle (rad) |
| 8 | angular velocity of second rotor | -Inf | Inf | ffoot | hinge | angle (rad) |
| 9 | x-coordinate of the front tip | -Inf | Inf | rootx | slide | velocity (m/s) |
| 10 | y-coordinate of the front tip | -Inf | Inf | rootz | slide | velocity (m/s) |
| 11 | angle of the front tip | -Inf | Inf | rooty | hinge | angular velocity (rad/s) |
| 12 | angle of the second rotor | -Inf | Inf | bthigh | hinge | angular velocity (rad/s) |
| 13 | angle of the second rotor | -Inf | Inf | bshin | hinge | angular velocity (rad/s) |
| 14 | velocity of the tip along the x-axis | -Inf | Inf | bfoot | hinge | angular velocity (rad/s) |
| 15 | velocity of the tip along the y-axis | -Inf | Inf | fthigh | hinge |angular velocity (rad/s) |
| 16 | angular velocity of front tip | -Inf | Inf | fshin | hinge | angular velocity (rad/s) |
| 17 | angular velocity of second rotor | -Inf | Inf | ffoot | hinge | angular velocity (rad/s) |
**Note:**
In practice (and Gym implementation), the first positional element is
omitted from the state space since the reward function is calculated based
on that value. This value is hidden from the algorithm, which in turn has
to develop an abstract understanding of it from the observed rewards.
Therefore, observation space has shape `(8,)` and looks like:
| Num | Observation | Min | Max | Name (in corresponding XML file) | Joint| Unit |
|-----|-----------------------|----------------------|--------------------|----------------------|--------------------|--------------------|
| 0 | y-coordinate of the front tip | -Inf | Inf | rootz | slide | position (m) |
| 0 | z-coordinate of the front tip | -Inf | Inf | rootz | slide | position (m) |
| 1 | angle of the front tip | -Inf | Inf | rooty | hinge | angle (rad) |
| 2 | angle of the second rotor | -Inf | Inf | bthigh | hinge | angle (rad) |
| 3 | angle of the second rotor | -Inf | Inf | bshin | hinge | angle (rad) |
Expand All @@ -95,37 +73,34 @@ class HalfCheetahEnv(mujoco_env.MujocoEnv, utils.EzPickle):
### Rewards
The reward consists of two parts:
- *reward_run*: A reward of moving forward which is measured
as *(x-coordinate before action - x-coordinate after action)/dt*. *dt* is
the time between actions and is dependeent on the frame_skip parameter
(default is 5), where the *dt* for one frame is 0.01 - making the
default *dt = 5*0.01 = 0.05*. This reward would be positive if the cheetah
runs forward (right) desired.
- *reward_control*: A negative reward for penalising the cheetah if it takes
actions that are too large. It is measured as *-coefficient x
sum(action<sup>2</sup>)* where *coefficient* is a parameter set for the
- *forward_reward*: A reward of moving forward which is measured
as *`forward_reward_weight` * (x-coordinate before action - x-coordinate after action)/dt*. *dt* is
the time between actions and is dependent on the frame_skip parameter
(fixed to 5), where the frametime is 0.01 - making the
default *dt = 5 * 0.01 = 0.05*. This reward would be positive if the cheetah
runs forward (right).
- *ctrl_cost*: A cost for penalising the cheetah if it takes
actions that are too large. It is measured as *`ctrl_cost_weight` *
sum(action<sup>2</sup>)* where *`ctrl_cost_weight`* is a parameter set for the
control and has a default value of 0.1
The total reward returned is ***reward*** *=* *reward_run + reward_control*
The total reward returned is ***reward*** *=* *forward_reward - ctrl_cost* and `info` will also contain the individual reward terms
### Starting State
All observations start in state (0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,) with a noise added to the
initial state for stochasticity. As seen before, the first 8 values in the
state are positional and the last 9 values are velocity. A uniform noise in
the range of [-0.1, 0.1] is added to the positional values while a standard
normal noise with a mean of 0 and standard deviation of 0.1 is added to the
the range of [-`reset_noise_scale`, `reset_noise_scale`] is added to the positional values while a standard
normal noise with a mean of 0 and standard deviation of `reset_noise_scale` is added to the
initial velocity values of all zeros.
### Episode Termination
The episode terminates when the episode length is greater than 1000.
### Arguments
No additional arguments are currently supported (in v2 and lower), but
modifications can be made to the XML file in the assets folder at
`gym/envs/mujoco/assets/half_cheetah.xml` (or by changing the path to a
modified XML file in another folder).
No additional arguments are currently supported in v2 and lower.
```
env = gym.make('HalfCheetah-v2')
Expand All @@ -137,6 +112,14 @@ class HalfCheetahEnv(mujoco_env.MujocoEnv, utils.EzPickle):
env = gym.make('HalfCheetah-v3', ctrl_cost_weight=0.1, ....)
```
| Parameter | Type | Default |Description |
|-------------------------|------------|----------------------|-------------------------------|
| `xml_file` | **str** | `"half_cheetah.xml"` | Path to a MuJoCo model |
| `forward_reward_weight` | **float** | `1.0` | Weight for *forward_reward* term (see section on reward) |
| `ctrl_cost_weight` | **float** | `0.1` | Weight for *ctrl_cost* weight (see section on reward) |
| `reset_noise_scale` | **float** | `0.1` | Scale of random perturbations of initial position and velocity (see section on Starting State) |
| `exclude_current_positions_from_observation`| **bool** | `True` | Whether or not to omit the x-coordinate from observations. Excluding the position can serve as an inductive bias to induce position-agnostic behavior in policies |
### Version History
Expand Down
Loading

0 comments on commit 95063a0

Please sign in to comment.