End-to-End Reinforcement Learning for Autonomous Driving

Visual-based Autonomous Driving Agent using PPO & SAC with Different Environments and Interpretability Analysis

Project Overview

This project implements an End-to-End Reinforcement Learning (RL) pipeline for autonomous driving using the Donkey Simulator(v25.10.06).

System Architecture

1. Image Process Pipelines

1. ROI Crop: Remove unnecessary sky/hood
2. Resize: Downsample to reduce computation
3. Random Lighting: Augment brightness (For Sim-to-Real)
4. Grayscale: Reduce dimensionality
5. Gaussian Noise: Augment sensor noise
6. Canny Edge: Structure extraction
7. Cutout: Augment occlusion robustness
8. Normalization: Scale to [0, 1]
9. Frame Stacking: Temporal information

2. Reinforcement Learning

• Action Space: Continuous control [Steer, Acceleration](Throttle when Acceleration > 0, Brake when less than 0).
• Reward Function:
  • Safety Elements
    • + Speed reward (keeping target velocity).
    • + progress reward (distance).
    • - Collision penalty.
    • - Lane Centering & Out-of-Lane penalty.
  • Comfort Elements
    • - Steering Smoothness penalty(restrict High-frequency oscillation)
    • - Acceleration Smoothness penalty
    • - Lateral Stability(Restrict Lateral Acceleration when high speed)

3. Interpretability (Post-Mortem Analysis)

• Triggers upon collision events.
• Generates Attention Maps using VisualBackProp [1], overlaying the last 4 seconds of input frames to visualize what the agent was focusing on (e.g., did it see the obstacle or was it looking at the clouds?).

---

Getting Started

1. clone this repository

  git clone https://github.com/Cerealmaster0621/e2e-autonomous-vehicle.git
  cd e2e-autonomous-vehicle

2. clone gym_donkeycar repository

  git clone https://github.com/Cerealmaster0621/gym-donkeycar.git
  cd gym-donkeycar
  pip install -e .
  cd ..

3. download DonkeySim simulator following the instruction inside the project folder

4. make conda environment and activate, install the requirements

  conda create -n e2e-autonomous-vehicle python=3.11 -y
  conda activate e2e-autonomous-vehicle
  pip install -r requirements.txt

5. (optional)run the test

  python src/test.py --sim {path to donkey_sim.app file}

Note: If you get permission errors, do not use sudo unless necessary.

---

[1] Bojarski, M. et al. (2017). VisualBackProp: efficient visualization of CNNs. arXiv preprint arXiv:1611.05418.