2022-04-27 Simulation Working Group meeting

[sglee-morai]

Useful Links

Previous Meeting Minutes

2022-04-13 Simulation Working Group meeting autowarefoundation/autoware.ai#2522

Issue Board

https://github.com/orgs/autowarefoundation/projects/7/views/1

General Discussion

E2E Simulator Requirements

Link for the discussion: #234

Interface with Autoware

• Tsutake-san has provided existing interfaces of TierIV Sim with Autoware, which can be used as a baseline
• no need to consider interface with autoware.ai

Environment & Scenario

• NOTE: Scenario requirement is discussed along with environment since the scenario has a dependency on the environment
• Still need to discuss more on: whether to create a 3d model of Hsinchu city or that of the ITRI campus
• HD map data for ITRI map & Hsinchu city are available in CI/CD Pipeline, Link will be added to the discussion page
• Tsutake-san suggested using random scenarios (Random traffics generated when a desired ego-vehicle path is given)
• No need to consider re-using scenarios created from TierIV Scenario Simulator (Integration is not available yet)
• However, it's good to have unit scenarios somewhat similar to those defined from ODD WG & TierIV Scenario Simulator
• For what should we use an E2E simulator?
  • Perception (e.g. vision-based sensor): This would be the primary purpose, but usage would not limited to this
  • Integration of the whole system
  • Control logic

Vehicle Model

• For a vehicle mode within the simulator to meet the performance requirement, testing within the simulator and tuning process would be required. For a starting point, a few variables can be fixed according to the reference model https://www.racev.com/product/k5 (a baseline model for ITRI)
• Who will create a vehicle model? TierIV and MORAI will have internal discussions and share opinions via Discord

GT(Ground Truth) & Metric Calculation

• For evaluation, it's essential for the E2E simulator to provide GT values
• Having a GT sensor would be a great feature
• In order to decide what values to be published from the simulator, it is first needed: which metrics are going to be calculated and how.
• Question: For bus-odd any metric calculation is already discussed?
  • Answer: currently scenarios that can run on the scenario simulator evaluate test results only as pass or fail
• [Further Discussion Needed] Need to discuss which metrics are going to be calculated for the bus-odd
• When it comes to positions, giving it w.r.t. world coordinate system would make the problem easier
• What is the metric from the localization node in Autoware? It is one from the NDT matching algorithm and the value represents how well the current estimate matches the reference point cloud map. Thus it is slightly different from the metrics that we would like to calculate using the E2E simulator.

Sensor

• Radar Simulation: Detailed use of radars for Autoware is still in discussion, thus requirements for E2E simulation are yet to be discussed
• The current sensor configuration on the ITRI bus is likely to be changed since the current configuration isn't compatible with the bus-odd feature in the Autoware.Universe

Additional Information about ITRI & ITRI's project status

• ITRI bus is currently based on Autoware.ai. ITRI should stick to the Autoware.ai for a few months more until the certification from the local government for driving the autonomous bus gets renewed. If they migrate Autoware.Universe beforehand, the certification process becomes much longer. Thus until ITRI successfully gets the certification renewed and migrate to the Autoware.Universe, ITRI wouldn't be able to directly integrate with E2E simulator.
• Aside from ITRI's own project, for AWF's own project, AWF asked ITRI whether ITRI can provide hardware and ECU for testing purposes. After maybe 2 months or so, Autoware.Universe can run on top of ITRI's hardware and ITRI campus. Thus, bus-odd codes in the Autoware.Universe will run anyway in 2 months or so.

Metric Calculation Node Development

Issue: autowarefoundation/autoware-projects#11

• Wojciech and his colleague in Robotec.ai (Domonik maybe?) is creating several generic metric calculation nodes (e.g. Localization algorithm statistics, Avg. velocity, ...)
• Some updates/merge requests are planned in 1 week (approx.)

Test Procedures to check Simulator's Compatibility with Autoware

Issue: autowarefoundation/autoware-projects#21

• Tsutake-san has provided existing interfaces of TierIV Sim with Autoware, which can be used as a baseline
• Sugwan is working on initial codes, which will be shared by the end of this week or early next week

Action Items

• Any members to keep adding on the simulator requirements, discussed here: Add synchronization system for perception #234
• Tsutake-san and Sugwan to have internal discussions respectively on creating a vehicle model
• Wojciech to keep working on Metric calculation node development autoware-projects#11 along with his colleague
• Sugwan to work on initial codes and documents for testing the simulator, and ask for feedback. (Define Test Procedures to check Simulator's Compatibility with Autoware autoware-projects#21)
• Sugwan to have internal discussions on creating environment models either for ITRI campus or Hsinchu city