-
Notifications
You must be signed in to change notification settings - Fork 7
Validation of NN packages
In 2019, two GSOC projects were conducted to benchmark neural network packages available on CRAN.
https://summerofcode.withgoogle.com/projects/#6319761942642688
https://summerofcode.withgoogle.com/projects/#5795523770974208
An initial list was prepared with 69 R packages having the “neural network” keyword either in the package title or in the package description: AMORE, ANN2, appnn, autoencoder, automl, BNN, brnn, Buddle, CaDENCE, cld2, cld3, condmixt, DALEX2, DamiaNN, deepnet, deepNN, DNMF, elmNNRcpp, ELMR, EnsembleBase, evclass, gamlss.add, gcForest, GMDH, GMDH2, GMDHreg, grnn, h2o, hybridEnsemble, isingLenzMC, keras, kerasformula, kerasR, leabRa, learNN, LilRhino, monmlp, neural, neuralnet, NeuralNetTools, NlinTS, nnet, nnetpredint, nnfor, onnx, OptimClassifier, OSTSC, pnn, predictoR, qrnn, QuantumOps, quarrint, radiant.model, rasclass, rcane, rminer, rnn, RSNNS, ruta, simpleNeural, snnR, softmaxreg, spnn, TeachNet, tensorflow, tfestimators, trackdem, TrafficBDE, tsensembler, validann.
This list was then reduced as we focused on neural networks for regression purpose. The awarded students developed files and tested 49 algorithms among 29 packages: AMORE, ANN2, automl, brnn, CaDENCE, deepnet, elmNNRcpp, ELMR, h2o, keras, MachineShop, minpack.lm, monmlp, neural, neuralnet, nlsr, nnet, qrnn, radiant.model, rcane, rminer, RSNNS, simpleNeural, snnR. The other packages were discarded because they are dedicated to classification, to convolutional networks or to other specialized topics unrelated with regression.
The 49 packages::algorithms were tested on 12 datasets provided by the mentors with different levels of difficulty. The metrics used to rank the algorithms were the root mean square error (RMSE) and the calculation duration. A summary table was obtained. For all algorithms and datasets, some X-Yobs+Ycalc figures (univariate datasets) or Yobs-Ycalc figures (multivariate datasets) were produced to display the result of the regression.
From the summary table, a global score was attributed to each package::algorithm. A first conclusion was the following: 12 packages::algorithms return good results and can be recommanded. 37 packages returned poor results and should be discarded. The datasets, the code, the results and the figures can be found here:
https://github.com/pkR-pkR/NNbenchmark
https://github.com/pkR-pkR/NNbenchmarkTemplates
https://akshajverma.com/NNbenchmarkWeb
Unfortunately, the above results were not published in a formal way as we fall short of time and found some glitches in our evaluation results that require some minor but time-consuming adjustements. For instance, several good packages use under the hood the same optim() function but with a different maximum for the number of iterations. This should be corrected as there is no reason to rank differently these similar algorithms.
Since the end of GSOC 2019, one mentor, Christophe Dutang, has rewritten the test code in a single R markdown file that treats all packages::algorithms in one run. This new code uses a generic part and one template per package. Although more complex, it offers a greater flexibility and permits the automatic export of data and graphic files. Furthermore, it incorporates a second convergence metrics to compare the algorithms. This code can be seen at :
https://github.com/pkR-pkR/NNbenchmarkTemplates
We believe the evaluation of all neural networks R packages related to regression will be completed by the end of this GSOC 2020. We were too optimistic in 2019.
We hope to attract a student with a sound knowledge in nonlinear regression algorithms (BFGS, Levenberg-Marquardt). The purpose of this work is to (1) to finalize the Rmd template, (2) incorporate new 2020 neural network packages, (3) incorporate R functions used the Rmd template in NNbenchmark, (4) optimize the NNbenchmark package, (5) submit NNbenchmark package on CRAN, (6) write a paper for the R journal and publish result on the website.
(1) The Rmd template needs to be finalized so that it generates results for all package over all datasets with the most comparable tuning parameters.
(2) Use RWsearch package to find new neural network packages and incorporate them into the template.
(3-4) The NNbenchmark package will be updated according to new development of the Rmd template as well as new packages tested. Optimization of this package should be performed so that benchmarks are quick, flexible, comparable and parallelized across datasets.
(5) Strong testing should be carried out to ensure that NNbenchmark meets the CRAN standards.
(6) The main format of publication of results is a publication in the R journal. Furthermore, an update of https://akshajverma.com/NNbenchmarkWeb/ will be performed.
With this work, we wish to alert R users about the varying performance of neural network packages. Users, both from academia and private companies, should be aware of the strengths and the weaknesses of the packages they use. We expect a bigger impact on package maintainers and authors and hope that such a benchmark will convince them to shift to better algorithms. Neural networks are understood as black boxes, especially nowadays with the advent of machine learning and artifical intelligence procedures, but a minimum of care and a sound mathematical approach should be taken when writing an R package.
Students, please contact mentors below after completing the tests below.
• Patrice Kiener is the author of the R packages FatTailsR and RWsearch and has 19 years of experience with neural networks of perceptron type. He managed the two previous GSOC projects in 2019.
• Christophe Dutang has authored or contributed to more than 10 packages and maintains the task views related to Probability Distributions and Extreme Value Analysis. He also had previous GSOC experience with the markovchain package in 2015 and 2016 and neural network in 2019.
• John Nash is retired Professor of Management in the Telfer School of Management, University of Ottawa. He has worked for approximately half a century in numerical methods for linear algebra and function minimization, in particular with R. See ‘optim()’ and packages such as optimx, Rvmmin, Rtnmin, Rcgmin, lbfgsb3 and nlsr.
Students, please do one or more of the following tests before contacting the mentors above.
• Use the code published in GOSC 2019 on a few datasets and comment the results.
• Have you ever contributed to an R package? if yes, which one?
• Have you ever contributed to a published article? if yes, which one?
Students, please send your test results at <gsoc2020_-+@-+_inmodelia.com> (redirects to P. Kiener and C. Dutang). List your answers to the above questions in the body of the email. For the R code, one attached -.R file at a maximum or one -.7z compressed file that concatenates a few -.R (-.Rmd) files.
Students can apply from February 20, 2020 to March 30, 2020, 18:00 UTC.