Distributed computing library implemented over PyCOMPSs programming model for HPC.

   

Website • Documentation • Releases • Slack

Introduction

The Distributed Computing Library (dislib) provides distributed algorithms ready to use as a library. So far, dislib is highly focused on machine learning algorithms, and it is greatly inspired by scikit-learn. However, other types of numerical algorithms might be added in the future. The library has been implemented on top of PyCOMPSs programming model, and it is being developed by the Workflows and Distributed Computing group of the Barcelona Supercomputing Center. dislib allows easy local development through docker. Once the code is finished, it can be run directly on any distributed platform without any further changes. This includes clusters, supercomputers, clouds, and containerized platforms. For more information on which infrastructures and architectures are supported refer to Availability.

Contents

• Quickstart
• Availability
• Contributing
• Citing dislib
• Acknowledgements
• License

Quickstart

Get started with dislib following our quickstart guide.

Availability

Currently, the following supercomputers have already PyCOMPSs installed and ready to use. If you need help configuring your own cluster or supercomputer, drop us an email and we will be pleased to help.

• Marenostrum 4 - Barcelona Supercomputing Center (BSC)
• Minotauro - Barcelona Supercomputing Center (BSC)
• Nord 3 - Barcelona Supercomputing Center (BSC)
• Cobi - Barcelona Supercomputing Center (BSC)
• Juron - Jülich Supercomputing Centre (JSC)
• Jureca - Jülich Supercomputing Centre (JSC)
• Ultraviolet - The Genome Analysis Center (TGAC)
• Archer - University of Edinburgh’s Advanced Computing Facility (ACF)
• Axiom - University of Novi Sad, Faculty of Sciences (UNSPMF)

Supported architectures:

• Intel SSF architectures
• IBM's Power 9

Contributing

Contributions are welcome and very much appreciated. We are also open to starting research collaborations or mentoring if you are interested in or need assistance implementing new algorithms. Please refer to our Contribution Guide for more details.

Citing dislib

If you use dislib in a scientific publication, we would appreciate you citing the following paper:

J. Álvarez Cid-Fuentes, S. Solà, P. Álvarez, A. Castro-Ginard, and R. M. Badia, "dislib: Large Scale High Performance Machine Learning in Python," in Proceedings of the 15th International Conference on eScience, 2019, pp. 96-105

Bibtex:

@inproceedings{dislib,
            title       = {{dislib: Large Scale High Performance Machine Learning in Python}},
            author      = {Javier Álvarez Cid-Fuentes and Salvi Solà and Pol Álvarez and Alfred Castro-Ginard and Rosa M. Badia},
            booktitle   = {Proceedings of the 15th International Conference on eScience},
            pages       = {96-105},
            year        = {2019},
 }            

Acknowledgements

This work has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement H2020-MSCA-COFUND-2016-754433.

This work has also received funding from the collaboration project between the Barcelona Supercomputing Center (BSC) and Fujitsu Ltd.

License

Apache License Version 2.0, see LICENSE