This is the project which @epiception, @SuhitK and I worked on for the Robot Localization and Mapping (16-833) course project at CMU in Spring 2019. The motivation of the project is to see how DeepVO can be enhanced using event-frames. To read more about event cameras and event SLAM, see this link.
This is a PyTorch implementation. This code has been tested on PyTorch 0.4.1 with CUDA 9.1 and CUDNN 7.1.2.
We have a dependency on a few packages, especially tqdm, scikit-image, tensorbordX and matplotlib. They can be installed using standard pip as below:
pip install scipy scikit-image matplotlib tqdm tensorboardX
To replicate the conda environment that we used for our training and evaluation
conda env create -f requirements.yml
The model assumes that FlowNet pre-trained weights are available for training. You can download the weights from @ClementPinard's implementation. Particularly, we need the weights for FlowNetS (flownets_EPE1.951.pth.tar). Instructions for downloadind the weights are in the README given there.
This model assumes the MVSEC datasets available from the Daniilidis Group at University of Pennsylvania. The code to sync the dataframes for event and intensity frames along with poses will be released soon.
To run the code for base DeepVO results - without any fusion, from the base directory of the repository, run:
sh exec.sh
To run the fusion model, from the base directory of the repository, run:
sh exec_fusion.sh