::: Neural network C++ implementations :::
Neurocl (Neuro Computing Library) was meant to experiment various 'from scratch' implementations of neural network layer schemes, focusing on matrix expression of feed forward/backward propagation steps. Initial release v0.1 only implemented Multi-Layer Perceptron (MLP) scheme, whereas v0.2 now incorporates a working Convolutional Neural Network (CONVNET) scheme.
Neurocl is C++11 compliant.
There are two different MLP implementations in Neurocl : one using standard Boost.ublas containers, and another one based on VexCL containers.
There is only one CONVNET implementation for now, based on a tensor abstraction class, using Boost.ublas containers.
As MLP was developped first, and emphasis was put on the CONVNET implementation for the past months, things are still managed a bit differently between these two architectures, but my plan is to factorize all possible things (solver, activation, loss function etc..) to have a common framework between them.
Please note that CONVNET implementation is still missing some configuration features and does not benefit of a fast tensor container implementation yet.
The upcoming evolutions will also focus on optimizations dedicated to running neural network based algorithms on small low power devices, like the ARM based Raspberry Pi.
I've been experimenting Neurocl on three main recognition/classification topics : mnist handwritten caracters recognition, automatic license plate recognition, and face recognition.
main library dependencies:
sample applications dependencies:
NOTE : there is a 3rd party software bootstrap script available for Raspbian:
$ sh bootstrap_3rdparty.shbuilding ccv (used for face detection):
$ git clone https://github.com/liuliu/ccv.git
$ cd ccv/lib
$ ./configure
$ makebuilding picoPi2 (used for Text-To-Speech on the Pi) and create an invokable TTS bash script:
$ git clone https://github.com/ch3ll0v3k/picoPi2.git
$ cd picoPi2/lib
$ make
$ cd ../tts
$ ./configure.py --make --2wav
$ echo -e '#!/bin/sh\nexport LD_LIBRARY_PATH="../lib"\n./picoPi2Wav "$1" -o speech.wav\naplay speech.wav' > speak.sh
$ sh speak.sh "This is a test"building neurocl (mainly header-only dependencies):
$ sudo apt-get install libboost-all-dev
$ git clone https://github.com/dtschump/CImg.git
$ git clone https://github.com/ddemidov/vexcl.git
$ git clone https://github.com/blackccpie/neurocl.git
$ cd neurocl
$ sh build_gcc_xxx.shNOTE: if experiencing problems due to libccv linking issues, rebuild the latter with the -fPIC compiler directive.
NOTE2: if experiencing gcc segfault problems when compiling on the raspberry pi, upgrade temporarily the swap file size to 100Mb (sudo nano /etc/dphys-swapfile).
neurocl requires three main input files:
-
the topology description file: this is a structured text file describing the neural net layers.
FOR MLP:
layer:0:28x28 layer:1:6x6 layer:2:10x1NOTE : MLP does not allow configurable activation functions for now, the default is sigmoid activations for all layers.
FOR CONVNET:
layer:in:0:28x28x1 layer:conv:1:24x24x3:5 layer:drop:2:24x24x3 layer:pool:3:12x12x3 layer:full:4:100x1x1 layer:out:5:10x1:1NOTE : CONVNET does not allow configurable activation functions for now, the default configuration is ReLU for convolutional layers, and Softmax with cross entropy error for the output layer. It can be edited in the src/convnet/network.cpp file.
-
the neural net weights file: this is a binary file containing the layers weight and bias values. This file is managed internally by neurocl, but user has to specify the name of the weights file to load for training/classifying.
-
the training set description file: this is a structured text file containing a list of image sample locations along with their expected output vectors. This file is only useful for net training steps.
/home/my_user/train-data/sample1.bmp 0 0 0 0 0 1 0 0 0 0 /home/my_user/train-data/sample2.bmp 1 0 0 0 0 0 0 0 0 0 /home/my_user/train-data/sample3.bmp 0 0 0 0 1 0 0 0 0 0 /home/my_user/train-data/sample4.bmp 0 1 0 0 0 0 0 0 0 0 /home/my_user/train-data/sample5.bmp 0 0 0 0 0 0 0 0 0 1 ... -
neurocl has basic xml configuration support: if present in the runtime directory, the neurocl.xml file will be parsed, but for now only a limited set of parameters are managed. The neurocl.xml file is formatted as shown below:
Example of MLP configuration file (default SGD solver):
<neurocl> <implementation>MLP</implementation> <learning_rate>1.5</learning_rate> </neurocl>
Example of CONVNET configuration file with a RMSPROP solver:
<neurocl> <implementation>CONVNET</implementation> <solver type="RMSPROP" lr="0.001" m="0.99"/> </neurocl>
neurocl main entry point is interface network_manager_interface, which can only be returned with the help of the factory class network_factory:
-
the network scheme can be built according to the xml configuration:
std::shared_ptr<network_manager_interface> net_manager = network_factory::build();
or given a specific scheme:
std::shared_ptr<network_manager_interface> net_manager = network_factory::build( network_factory::t_neural_impl::NEURAL_IMPL_CONVNET );The two availables schemes are:
-
NEURAL_IMPL_MLP
3 backends available:
- NEURAL_IMPL_BNU_REF : the reference implementation only using boost::numeric::ublas containers and operators.
- NEURAL_IMPL_BNU_FAST : experimental fast implementation using boost::numeric::ublas containers but custom simd (neon/sse4) optimized operators (for now layer sizes should be multiples of 4).
- NEURAL_IMPL_VEXCL : experimental vexcl reference implementation.
-
NEURAL_IMPL_CONVNET
2 backends available:
- CONVNET : the single-threaded implementation.
- CONVNET_PARALLEL : the multi-threaded implementation (only for training for now).
Note1 : MLP default backend is hardcoded to NEURAL_IMPL_BNU_REF.
Note2 : CONVNET default backend is hardcoded to CONVNET.
Note3 : these hardcoded settings can be changed in the network_factory class.
-
-
a given network can be loaded, given its topology and weights file names
net_manager->load_network( "topology.txt", "weights.bin" );
-
once a network is loaded, it can be trained with the help of the samples_manager class:
samples_manager smp_train_manager; smp_train_manager.load_samples( "../nets/mnist/training/mnist-train.txt" ); net_manager->batch_train( smp_train_manager, NB_EPOCHS, BATCH_SIZE );
-
or used for direct output computation:
neurocl::sample sample(...); net_manager->compute_output( sample );
-
The reference sample application to look for best practice code is mnist_autotrainer, located in the apps directory.
The mnist_autotrainer application also illustrates dumping training data in a dedicated .csv file. This data file can be graphically represented in a web page using CanvasJS framework (as shown at the top of this doc), ie by running a lightweight python webserver and viewing the right url:
$ cd neurocl
$ python -m SimpleHTTPServer
$ firefox http://127.0.0.1:8000/webNOTE : for the sake of clarity, the following diagrams have been simplified to the main classes. More detailed diagrams will be added as appendices soon.
MLP Class diagram:
CONVNET Class diagram:
- Mac OSX
- Debian Jessie
- Ubuntu 16.04 LTS
- Raspberry Pi Raspbian
MLP:
CONVNETS:
CLASSIFIERS:
SOLVERS:
FRENCH SPEAKING:
- http://clement.chatelain.free.fr/enseignements/rdn.pdf
- http://emilie.caillault.free.fr/doc/EPoisson2001.pdf
- http://page.mi.fu-berlin.de/rojas/neural/chapter/K7.pdf -> ß7.3.3 concerning matrix form
neurocl source is distributed under MIT License