0.8.0 changes -> master (Unity-Technologies#273)

* Fix HDRP warnings on 2020.X (Unity-Technologies#260) * Fix HDRP warnings on 2020.2 * Updating changelog * Fixing HDRP version expression on 2019.4 * Fix keypoint issues (Unity-Technologies#259) * Fixing two bugs in KeypointLabeler: AISV-1254 - Some keypoints fall outside of the image bounds AISV-1257 - Partially visible characters with no visible keypoints are not reported * Adding JointLabelEditor with message about Labeling requirement * Reorganizing keypoint docs and elaborating on setup requirements. * Updating PerceptionCamera.md * Fixing issues on 2020.2 and failing test * Add object filtering modes to keypoint labeler * Updating docs for object filtering * Adding link to tutorial * Moving KeypointLabeler.md to proper directory * Update TUTORIAL.md * Updating capitalization Co-authored-by: Mohsen K <[email protected]> * Apply doc suggestions Co-authored-by: Mohsen K <[email protected]> * Update PerceptionCamera.md based on feedback * Update CHANGELOG.md Co-authored-by: Mohsen K <[email protected]> * Merge master to 0.8.0 (Unity-Technologies#265) * addressed feedback - new test for keypoint labeler - added ClearMaterialProperties method to IGroundTruthGenerator - Other fixes * docs clarification * Update CHANGELOG.md * added missing interface method implementation for HDRP * Amending dataset schema to describe correct coordinate system (Unity-Technologies#255) The summary for the 3d bounding box states the coordinates are from the sensor's reference frame, but the string for the translational component says it's global. I *think* this is just a typo in the doc, so correcting the description to match the summary for this type of entry. * Adding supported version badges (Unity-Technologies#256) * Cloud rendering support (Unity-Technologies#227) * Adding support for cloud rendering in the run simulation window * minor fixes * minor update * Update BoundingBox3DLabeler.cs * Update BoundingBox3DLabeler.cs * inner mesh scaling and disabled renderes * A fix that prevents normal sampler returning values outside of min max (Unity-Technologies#258) * Update pull_request_template.md (Unity-Technologies#257) * Update com.unity.perception/CHANGELOG.md * Increasing color variety in instance segmentation. (Unity-Technologies#261) * Increasing color variety in instance segmentation. * Fixing duplicated color issues in InstanceIdToColorMapping introduced after expanding range. * Using other constants in constant for cleanliness Co-authored-by: Mohsen Kamalzadeh <> Co-authored-by: Wesley Smith <[email protected]> Co-authored-by: Mohsen K <[email protected]> Co-authored-by: Devin Miller (Unity) <[email protected]> Co-authored-by: PriyeshWani <[email protected]> Co-authored-by: Steve Borkman <[email protected]> * fixed bracket spacing in KeypointLabeler * Add 3dbb tests (Unity-Technologies#267) * Adding and updating 3d bb tests. * Adding more BB3D tests * Adding more tests for camera transforms and fixing labeler. * Added tooltips to sample randomizers * Fix warnings caused by animation randomizer running on disabled GameObjects (Unity-Technologies#268) * Updating packge change log and version Co-authored-by: Mohsen K <[email protected]> Co-authored-by: Wesley Smith <[email protected]> Co-authored-by: Devin Miller (Unity) <[email protected]> Co-authored-by: PriyeshWani <[email protected]> Co-authored-by: Steve Borkman <[email protected]> Co-authored-by: sleal-unity <[email protected]> Co-authored-by: Mohsen Kamalzadeh <>
ooMike1 · Mar 26, 2021 · 582516e · 582516e
1 parent c4de1b7
commit 582516e
Show file tree

Hide file tree

Showing 27 changed files with 792 additions and 158 deletions.
diff --git a/TestProjects/PerceptionHDRP/Packages/packages-lock.json b/TestProjects/PerceptionHDRP/Packages/packages-lock.json
@@ -81,7 +81,7 @@
         "com.unity.collections": "0.9.0-preview.6",
         "com.unity.nuget.newtonsoft-json": "1.1.2",
         "com.unity.render-pipelines.core": "7.1.6",
-        "com.unity.simulation.capture": "0.0.10-preview.19",
+        "com.unity.simulation.capture": "0.0.10-preview.20",
         "com.unity.simulation.client": "0.0.10-preview.10",
         "com.unity.simulation.core": "0.0.10-preview.22"
       }
@@ -126,7 +126,7 @@
       "url": "https://packages.unity.com"
     },
     "com.unity.simulation.capture": {
-      "version": "0.0.10-preview.19",
+      "version": "0.0.10-preview.20",
       "depth": 1,
       "source": "registry",
       "dependencies": {

diff --git a/com.unity.perception/CHANGELOG.md b/com.unity.perception/CHANGELOG.md
@@ -22,7 +22,19 @@ Increased color variety in instance segmentation images
 
 ### Fixed
 
-A bug in the Normal Sampler where it would return values less than the passed in min, or greater than the passed in max, for random values very close to 0 or 1 respectively.
+## [0.8.0-preview.3] - 2021-03-24
+### Changed
+
+Expanded documentation on the Keypoint Labeler
+Updated Keypoint Labeler logic to only report keypoints for visible objects by default
+Increased color variety in instance segmentation images
+
+### Fixed
+
+Fixed compiler warnings in projects with HDRP on 2020.1 and later
+
+Fixed a bug in the Normal Sampler where it would return values less than the passed in minimum value, or greater than the passed in maximum value, for random values very close to 0 or 1 respectively.
+
 
 ## [0.8.0-preview.2] - 2021-03-15
 

diff --git a/com.unity.perception/Documentation~/GroundTruth/KeypointLabeler.md b/com.unity.perception/Documentation~/GroundTruth/KeypointLabeler.md
@@ -0,0 +1,91 @@
+# Keypoint Labeler
+
+The Keypoint Labeler captures the screen locations of specific points on labeled GameObjects. The typical use of this Labeler is capturing human pose estimation data, but it can be used to capture points on any kind of object. The Labeler uses a [Keypoint Template](#KeypointTemplate) which defines the keypoints to capture for the model and the skeletal connections between those keypoints. The positions of the keypoints are recorded in pixel coordinates.
+
+## Data Format
+The keypoints captured each frame are in the following format:
+```
+keypoints {
+  label_id:      <int>   -- Integer identifier of the label
+  instance_id:   <str>   -- UUID of the instance.
+  template_guid: <str>   -- UUID of the keypoint template
+  pose:          <str>   -- Current pose
+  keypoints [            -- Array of keypoint data, one entry for each keypoint defined in associated template file.
+    {
+      index:     <int>   -- Index of keypoint in template
+      x:         <float> -- X pixel coordinate of keypoint
+      y:         <float> -- Y pixel coordinate of keypoint
+      state:     <int>   -- Visibility state
+    }, ...
+  ]
+}
+```
+
+The `state` entry has three possible values: 
+* 0 - the keypoint either does not exist or is outside of the image's bounds
+* 1 - the keypoint exists inside of the image bounds but cannot be seen because the object is not visible at its location in the image
+* 2 - the keypoint exists and the object is visible at its location
+
+The annotation definition, captured by the Keypoint Labeler once in each dataset, describes points being captured and their skeletal connections. These are defined by the [Keypoint Template](#KeypointTemplate).
+```
+annotation_definition.spec {
+  template_id:       <str>           -- The UUID of the template
+  template_name:     <str>           -- Human readable name of the template
+  key_points [                       -- Array of joints defined in this template
+    {
+      label:         <str>           -- The label of the joint
+      index:         <int>           -- The index of the joint
+    }, ...
+  ]
+  skeleton [                         -- Array of skeletal connections (which joints have connections between one another) defined in this template
+    {
+      joint1:        <int>           -- The first joint of the connection
+      joint2:        <int>           -- The second joint of the connection
+    }, ...
+  ]
+}
+```
+
+## Setup
+The Keypoint Labeler captures keypoints each frame from each object in the scene that meets the following conditions:
+
+* The object or its children are at least partially visible in the frame 
+  * The _Object Filter_ option on the Keypoint Labeler can be used to also include fully occluded or off-screen objects
+* The root object has a `Labeling` component
+* The object matches at least one entry in the Keypoint Template by either:
+  * Containing an Animator with a [humanoid avatar](https://docs.unity3d.com/Manual/ConfiguringtheAvatar.html) whose rig matches a keypoint OR
+  * Containing children with Joint Label components whose labels match keypoints
+
+For a tutorial on setting up your project for keypoint labeling, see the [Human Pose Labeling and Randomization Tutorial](../HPTutorial/TUTORIAL.md).
+
+## Keypoint Template
+
+Keypoint Templates are used to define the keypoints and skeletal connections captured by the Keypoint Labeler. The Keypoint Template takes advantage of Unity's humanoid animation rig, and allows the user to automatically associate template keypoints to animation rig joints. Additionally, the user can choose to ignore the rigged points, or add points not defined in the rig.
+
+A [COCO](https://cocodataset.org/#home) Keypoint Template is included in the Perception package.
+
+### Editor
+
+The Keypoint Template editor allows the user to create/modify a Keypoint Template. The editor consists of the header information, the keypoint array, and the skeleton array.
+
+![Header section of the keypoint template](../images/keypoint_template_header.png)
+<br/>_Header section of the keypoint template_
+
+In the header section, a user can change the name of the template and supply textures that they would like to use for the keypoint visualization.
+
+![The keypoint section of the keypoint template](../images/keypoint_template_keypoints.png)
+<br/>_Keypoint section of the keypoint template_
+
+The keypoint section allows the user to create/edit keypoints and associate them with Unity animation rig points. Each keypoint record
+has 4 fields: label (the name of the keypoint), Associate to Rig (a boolean value which, if true, automatically maps the keypoint to
+the GameObject defined by the rig), Rig Label (only needed if Associate To Rig is true, defines which rig component to associate with
+the keypoint), and Color (RGB color value of the keypoint in the visualization).
+
+![Skeleton section of the keypoint template](../images/keypoint_template_skeleton.png)
+<br/>_Skeleton section of the keypoint template_
+
+The skeleton section allows the user to create connections between joints, basically defining the skeleton of a labeled object.
+
+#### Animation Pose Label
+
+This file is used to define timestamps in an animation to a pose label.
diff --git a/com.unity.perception/Documentation~/HPTutorial/TUTORIAL.md b/com.unity.perception/Documentation~/HPTutorial/TUTORIAL.md
@@ -248,7 +248,7 @@ You can now check out the output dataset to see what the annotations look like.
 }
 ```
 
-In the above annotation, all of the 18 joints defined in the COCO template we used are listed. For each joint that is present in our character, you can see the X and Y coordinates within the captured frame. However, you may notice three of the joints are listed with (0,0) coordinates. These joints are not present in our character. A fact that is also denoted by the `state` field. A state of **0** means the joint was not present, **1** denotes a joint that is present but not visible (to be implemented in a later version of the package), and **2** means the joint was present and visible.
+In the above annotation, all of the 18 joints defined in the COCO template we used are listed. For each joint that is present in our character, you can see the X and Y coordinates within the captured frame. However, you may notice three of the joints are listed with (0,0) coordinates. These joints are not present in our character. A fact that is also denoted by the `state` field. A state of **0** means the joint either does not exist or is outside of the image's bounds, **1** denotes a joint that is inside of the image but cannot be seen because the part of the object it belongs to is not visible in the image, and **2** means the joint was present and visible.
 
 You may also note that the `pose` field has a value of `unset`. This is because we have not defined poses for our animation clip and `Perception Camera` yet. We will do this next.
 

diff --git a/com.unity.perception/Documentation~/PerceptionCamera.md b/com.unity.perception/Documentation~/PerceptionCamera.md
@@ -91,83 +91,13 @@ The ObjectCountLabeler records object counts for each label you define in the Id
 ```
 _Example rendered object info for a single object_
 
-The RenderedObjectInfoLabeler records a list of all objects visible in the Camera image, including its instance ID, resolved label ID and visible pixels. If Unity cannot resolve objects to a label in the IdLabelConfig, it does not record these objects.
+The `RenderedObjectInfoLabeler` records a list of all objects visible in the camera image, including their instance IDs, resolved label IDs, and visible pixel counts. If Unity cannot resolve objects to a label in the `IdLabelConfig`, it does not record these objects.
 
 ### Keypoint Labeler
 
-The keypoint labeler captures keypoints of a labeled gameobject. The typical use of this labeler is capturing human pose estimation data. The labeler uses a [keypoint template](#KeypointTemplate) which defines the keypoints to capture for the model and the skeletal connections between those keypoints. The positions of the keypoints are recorded in pixel coordinates. Each keypoint has a state value: 0 - the keypoint either does not exist or is outside of the image's bounds, 1 - the keypoint exists and is inside of the image's bounds but cannot be seen because it is occluded by another object, and 2 - the keypoint exists and is visible.
+The keypoint labeler captures the screen locations of specific points on labeled GameObjects. The typical use of this Labeler is capturing human pose estimation data, but it can be used to capture points on any kind of object. The Labeler uses a [Keypoint Template](#KeypointTemplate) which defines the keypoints to capture for the model and the skeletal connections between those keypoints. The positions of the keypoints are recorded in pixel coordinates.
 
-```
-keypoints {
-  label_id:      <int>   -- Integer identifier of the label
-  instance_id:   <str>   -- UUID of the instance.
-  template_guid: <str>   -- UUID of the keypoint template
-  pose:          <str>   -- Pose ground truth information
-  keypoints [            -- Array of keypoint data, one entry for each keypoint defined in associated template file.
-    {
-      index:     <int>   -- Index of keypoint in template
-      x:         <float> -- X pixel coordinate of keypoint
-      y:         <float> -- Y pixel coordinate of keypoint
-      state:     <int>   -- 0: keypoint does not exist, 1: keypoint exists but is not visible, 2: keypoint exists and is visible
-    }, ...
-  ]
-}
-```
-
-#### Keypoint Template
-
-keypoint templates are used to define the keypoints and skeletal connections captured by the KeypointLabeler. The keypoint
-template takes advantage of Unity's humanoid animation rig, and allows the user to automatically associate template keypoints
-to animation rig joints. Additionally, the user can choose to ignore the rigged points, or add points not defined in the rig.
-A Coco keypoint template is included in the perception package.
-
-##### Editor
-
-The keypoint template editor allows the user to create/modify a keypoint template. The editor consists of the header information,
-the keypoint array, and the skeleton array.
-
-![Header section of the keypoint template](images/keypoint_template_header.png)
-<br/>_Header section of the keypoint template_
-
-In the header section, a user can change the name of the template and supply textures that they would like to use for the keypoint
-visualization.
-
-![The keypoint section of the keypoint template](images/keypoint_template_keypoints.png)
-<br/>_Keypoint section of the keypoint template_
-
-The keypoint section allows the user to create/edit keypoints and associate them with Unity animation rig points. Each keypoint record
-has 4 fields: label (the name of the keypoint), Associate to Rig (a boolean value which, if true, automatically maps the keypoint to
-the gameobject defined by the rig), Rig Label (only needed if Associate To Rig is true, defines which rig component to associate with
-the keypoint), and Color (RGB color value of the keypoint in the visualization).
-
-![Skeleton section of the keypoint template](images/keypoint_template_skeleton.png)
-<br/>_Skeleton section of the keypoint template_
-
-The skeleton section allows the user to create connections between joints, basically defining the skeleton of a labeled object.
-
-##### Format
-```
-annotation_definition.spec {
-  template_id:       <str>           -- The UUID of the template
-  template_name:     <str>           -- Human readable name of the template
-  key_points [                       -- Array of joints defined in this template
-    {
-      label:         <str>           -- The label of the joint
-      index:         <int>           -- The index of the joint
-    }, ...
-  ]
-  skeleton [                         -- Array of skeletal connections (which joints have connections between one another) defined in this template
-    {
-      joint1:        <int>           -- The first joint of the connection
-      joint2:        <int>           -- The second joint of the connection
-    }, ...
-  ]
-}
-```
-
-#### Animation Pose Label
-
-This file is used to define timestamps in an animation to a pose label.
+For more information, see [Keypoint Labeler](GroundTruth/KeypointLabeler.md) or the [Human Pose Labeling and Randomization Tutorial](HPTutorial/TUTORIAL.md)
 
 ## Limitations
 

diff --git a/com.unity.perception/Editor/GroundTruth/JointLabelEditor.cs b/com.unity.perception/Editor/GroundTruth/JointLabelEditor.cs
@@ -0,0 +1,22 @@
+using System.Linq;
+using UnityEngine;
+using UnityEngine.Perception.GroundTruth;
+
+namespace UnityEditor.Perception.GroundTruth
+{
+    [CustomEditor(typeof(JointLabel))]
+    public class JointLabelEditor : Editor
+    {
+        public override void OnInspectorGUI()
+        {
+            base.OnInspectorGUI();
+#if UNITY_2020_1_OR_NEWER
+            //GetComponentInParent<T>(bool includeInactive) only exists on 2020.1 and later
+            if (targets.Any(t => ((Component)t).gameObject.GetComponentInParent<Labeling>(true) == null))
+#else
+            if (targets.Any(t => ((Component)t).GetComponentInParent<Labeling>() == null))
+#endif
+                EditorGUILayout.HelpBox("No Labeling component detected on parents. Keypoint labeling requires a Labeling component on the root of the object.", MessageType.Info);
+        }
+    }
+}
diff --git a/com.unity.perception/Editor/GroundTruth/JointLabelEditor.cs.meta b/com.unity.perception/Editor/GroundTruth/JointLabelEditor.cs.meta
diff --git a/com.unity.perception/Runtime/GroundTruth/Labelers/BoundingBox3DLabeler.cs b/com.unity.perception/Runtime/GroundTruth/Labelers/BoundingBox3DLabeler.cs
@@ -294,14 +294,13 @@ void ProcessLabel(Labeling labeledEntity)
                     combinedBounds.center = labelTransform.TransformPoint(combinedBounds.center);
                     combinedBounds.extents = Vector3.Scale(combinedBounds.extents,  labelTransform.lossyScale);
 
-                    // Now convert all points into camera's space
-                    var cameraCenter = cameraTransform.InverseTransformPoint(combinedBounds.center);
-                    cameraCenter = Vector3.Scale(cameraTransform.localScale, cameraCenter);
+                    // Now adjust the center and rotation to camera space. Camera space transforms never rescale objects
+                    combinedBounds.center = combinedBounds.center - cameraTransform.position;
+                    combinedBounds.center = Quaternion.Inverse(cameraTransform.rotation) * combinedBounds.center;
 
-                    // Rotation to go from label space to camera space
                     var cameraRotation = Quaternion.Inverse(cameraTransform.rotation) * labelTransform.rotation;
 
-                    var converted = ConvertToBoxData(labelEntry, labeledEntity.instanceId, cameraCenter, combinedBounds.extents, cameraRotation);
+                    var converted = ConvertToBoxData(labelEntry, labeledEntity.instanceId, combinedBounds.center, combinedBounds.extents, cameraRotation);
 
                     m_BoundingBoxValues[m_CurrentFrame][labeledEntity.instanceId] = converted;
                 }
@@ -311,7 +310,7 @@ void ProcessLabel(Labeling labeledEntity)
         static Vector3 CalculateRotatedPoint(Camera cam, Vector3 start, Vector3 xDirection, Vector3 yDirection, Vector3 zDirection, float xScalar, float yScalar, float zScalar)
         {
             var rotatedPoint = start + xDirection * xScalar + yDirection * yScalar + zDirection * zScalar;
-            var worldPoint = cam.transform.TransformPoint(rotatedPoint);
+            var worldPoint = cam.transform.position + cam.transform.rotation * rotatedPoint;
             return VisualizationHelper.ConvertToScreenSpace(cam, worldPoint);
         }
-Original file line number
+Diff line change
@@ Expand Up @@
     }
     ```
-    In the above annotation, all of the 18 joints defined in the COCO template we used are listed. For each joint that is present in our character, you can see the X and Y coordinates within the captured frame. However, you may notice three of the joints are listed with (0,0) coordinates. These joints are not present in our character. A fact that is also denoted by the `state` field. A state of **0** means the joint was not present, **1** denotes a joint that is present but not visible (to be implemented in a later version of the package), and **2** means the joint was present and visible.
+    In the above annotation, all of the 18 joints defined in the COCO template we used are listed. For each joint that is present in our character, you can see the X and Y coordinates within the captured frame. However, you may notice three of the joints are listed with (0,0) coordinates. These joints are not present in our character. A fact that is also denoted by the `state` field. A state of **0** means the joint either does not exist or is outside of the image's bounds, **1** denotes a joint that is inside of the image but cannot be seen because the part of the object it belongs to is not visible in the image, and **2** means the joint was present and visible.
     You may also note that the `pose` field has a value of `unset`. This is because we have not defined poses for our animation clip and `Perception Camera` yet. We will do this next.
@@ Expand Down @@