This repository contains a custom-built Unity simulation environment and Reinforcement Learning (RL) architecture for the NeoRacer autonomous platform. Moving away from standard autograders and pre-built tracks, this environment is built from the ground up to streamline the ML-Agents training process.
Our RL model utilizes Proximal Policy Optimization (PPO) combined with Imitation Learning to navigate the track autonomously.
- Sensor Suite (Custom LiDAR): Transitioned from heavy, high-density LiDAR to a highly optimized, downsampled Ray Perception Sensor 3D (Raycast) array to reduce computational overhead and prevent vector space mismatches.
- Reward Function: Replaced sparse, discrete checkpoint rewards with a continuous, dense gradient. The step quality is calculated by multiplying normalized speed and alignment vectors, smoothed via the Power Mean to prevent erratic agent behavior.
- Training Pipeline: Utilizes Behavioral Cloning (BC) for the initial training phase to establish an optimal racing line, automatically transitioning to pure RL for time-optimization and trajectory refinement.
For deep dives into the math, setup, and research, please refer to our documentation folder:
- 📖 Student Training Guide: Step-by-step tutorial on recording demonstrations, editing YAML configs, and reading TensorBoard.
- 🔬 RL Architecture & Reward Math: Detailed breakdown of the continuous reward function, failsafes, and sensor downsampling.
- 🔗 References & Research: Academic literature and code references (including MIT simulation origins).
- Install Nix using the Determinate Systems installer:
curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install
- Log out and back in to ensure Nix is properly initialized
- Clone this repository and cd to that directory
- Enter the development environment:
nix develop
- Build the simulation:
make
-
Clone this repository and cd to that directory.
-
Create conda envrionment on that folder to keep the python side consistant:
conda env create -f environment.yml && conda activate mlagentsNote: when you start training, use
conda activate mlagentsto activate the envrionment. -
For Mac user: in your mlagents envrionment, first do
pip3 install grpcioand thenpython -m pip install mlagents==1.1.0You should be able to run
mlagents-learn --helpin the conda envrionment. -
Open the repository in Unity. Select the Unity version as suggested. This ensures that the version control won't get messed up.
-
After you opened it, go to "File"-->"Build and Run" to build the app. Alternatively, you can press "Command + B" for this step.
You can run the simulation in two ways:
- Run the built simulation directly:
./Builds/sim
- Run with Python control:
cd python && python gaussianForNewSim.py
Use command mlagents-learn {NNParameter.yaml} --run-id={a unique name for this training session}
Note: If you have to quit (Ctrl-C) before it finishes training, you can run pass in --resume flag to the command. mlagents-learn {NNParameter.yaml} --run-id={a unique name for this training session} --resume
When the message "Start training by pressing the Play button in the Unity Editor" is displayed on the screen, you can press the Play button in Unity to start training in the Editor.
Observe the Training:
Run tensorboard --logdir results then go to localhost:6006 in your browser.
Initial setup includes codes from MIT simulation:
- Scripts folder: CenterOfMass.cs
- Scripts/Racecar folder: Racecar.cs, RacecarModule,cs, RacecarNWH.cs, Drive.cs, PhysicsModule.cs, Lidar.cs and CameraModule.cs
- Scripts/Static folder: Constants.cs, NormalDist.cs, Settings.csScripts/UI folder: Hud.cs, ScreenManger.cs
Modification on those files were later adapted.