feat(fix): fix bevfusion deployment script #98
Conversation
SamratThapa120
changed the title
|
|
||
| feats = feats.sum(dim=1, keepdim=False) / sizes.type_as(feats).view(-1, 1) | ||
| # feats = batch_inputs_dict["voxels"]["voxels"] |
| @@ -15,8 +15,8 @@ | |||
| model_inputs=[ | |||
| dict( | |||
| input_shapes=dict( | |||
| voxels=dict(min_shape=[1, 5], opt_shape=[64000, 5], max_shape=[256000, 5]), | |||
| coors=dict(min_shape=[1, 4], opt_shape=[64000, 4], max_shape=[256000, 4]), | |||
| voxels=dict(min_shape=[1, 10, 4], opt_shape=[64000, 10, 4], max_shape=[256000, 10, 4]), | |||
There was a problem hiding this comment.
The shape should be [M, maximum number of points, features], which it will be [M, 10, 5] if we are using intensity right?
| export_params=True, | ||
| input_names=input_names, | ||
| output_names=output_names, | ||
| opset_version=opset_version, | ||
| dynamic_axes=dynamic_axes, | ||
| keep_initializers_as_inputs=keep_initializers_as_inputs, | ||
| verbose=verbose, | ||
| do_constant_folding=False, |
There was a problem hiding this comment.
why disable though? I believe the constant_folding can speed up inference
| @@ -18,7 +21,35 @@ | |||
| voxels=dict(min_shape=[1, 10, 4], opt_shape=[64000, 10, 4], max_shape=[256000, 10, 4]), | |||
| coors=dict(min_shape=[1, 3], opt_shape=[64000, 3], max_shape=[256000, 3]), | |||
| num_points_per_voxel=dict(min_shape=[1], opt_shape=[64000], max_shape=[256000]), | |||
| image_feats=dict(min_shape=[80, 180, 180], opt_shape=[80, 180, 180], max_shape=[80, 180, 180]), | |||
| # TODO(TIERIV): Optimize. Now, using points will increase latency significantly | |||
There was a problem hiding this comment.
It's better to put TODO(YourName)
There was a problem hiding this comment.
Is it because we want to compute depth map so we need to input points? This will actually run internal voxelization again in here
https://github.com/tier4/AWML/blob/main/projects/BEVFusion/bevfusion/bevfusion.py#L172
| @@ -18,7 +21,35 @@ | |||
| voxels=dict(min_shape=[1, 10, 4], opt_shape=[64000, 10, 4], max_shape=[256000, 10, 4]), | |||
| coors=dict(min_shape=[1, 3], opt_shape=[64000, 3], max_shape=[256000, 3]), | |||
| num_points_per_voxel=dict(min_shape=[1], opt_shape=[64000], max_shape=[256000]), | |||
| image_feats=dict(min_shape=[80, 180, 180], opt_shape=[80, 180, 180], max_shape=[80, 180, 180]), | |||
| # TODO(TIERIV): Optimize. Now, using points will increase latency significantly | |||
There was a problem hiding this comment.
Is it because we want to compute depth map so we need to input points? This will actually run internal voxelization again in here
https://github.com/tier4/AWML/blob/main/projects/BEVFusion/bevfusion/bevfusion.py#L172
This pull request introduces several improvements to the BEVFusion deployment pipeline, particularly for ONNX export. The changes focus on refactoring the model's forward and feature extraction logic to better handle image features, updating deployment configs for more flexible input shapes, and fixing geometry computation in the depth module. These updates enhance the modularity and exportability of the BEVFusion model, facilitating more efficient deployment and inference.
Model code refactoring and ONNX export support:
The bevfusion-cl model is split into two parts in deployment:
image_backboneandmain_body. Theimage_backbonetakes the image input and outputs the features, themain_bodydoes the rest of the predictions.Removed the old deployment scripts.
Refactored
BEVFusionmodel methods (_forward,predict,loss, andextract_feat) to consistently accept a newusing_image_featuresflag, improving control over whether to use precomputed image features during inference or export. Added a newget_image_backbone_featureshelper to modularize image feature extraction. [1] [2] [3] [4] [5] [6] [7] [8] [9]Updated the logic in
extract_featto handle ONNX inference mode, including geometry feature handling and disabling point features when necessary. [1] [2]Deployment configuration updates:
Updated
bevfusion_camera_backbone_tensorrt_dynamic.pyandbevfusion_main_body_with_image_tensorrt_dynamic.pyto use more flexible and parameterized input shapes (e.g.,image_dims,depth_bins,feature_dims). Expanded the set of dynamic axes and input names to support a wider range of deployment scenarios, and added missing inputs required for the main body with image features. [1] [2] [3] [4] [5] [6]Adjusted
bevfusion_main_body_lidar_only_tensorrt_dynamic.pyto update the expected voxel and coordinate shapes for lidar-only inference.Depth module geometry computation fix:
depth_lss.py, ensuring that transformations and matrix inverses are only computed when not using precomputed geometry features. [1] [2]Documentation updates:
README.mddeployment command examples to include the--moduleargument, clarifying how to export specific model components. [1] [2]Note to reviewer.
Please try running the deployment script, using models included in this PR #88