Skip to content
This repository was archived by the owner on Jun 12, 2023. It is now read-only.

Building TileDB

Jump to bottom
Stavros Papadopoulos edited this page Jan 26, 2017 · 95 revisions
# Requirements

TileDB has been tested on Ubuntu Linux (v.15.10), CentOS Linux (v.7.3.1611) and Mac OS X El Capitan (v.10.11.2), but TileDB should work with any reasonably recent version of Ubuntu, CentOS or Mac OS X. Moreover, it has been tested with g++ v.5.3.0 on Ubuntu, g++ v.4.8.5 on CentOS, and Clang Apple LLVM version 7.3.0 on Mac OS X.

Library Dependencies

TileDB has minimal dependencies, relying only on the following standard libraries:

Required dependencies:

Optional dependencies:

  • MPI (for multi-processing operations)
  • OpenMP (for multi-threaded operations)
  • Google Test (for code testing)
  • Doxygen (for code documentation)
# Build on Ubuntu Linux

Step 1 - Install cmake, wget and git

You first need to install three useful programs, cmake, GNU wget and git. Simply run:

sudo apt-get install cmake
sudo apt-get install wget
sudo apt-get install git

Step 2 - Install zlib, OpenSSL, LZ4 and MPI

Simply run:

sudo apt-get install zlib1g-dev
sudo apt-get install libssl-dev
sudo apt-get install liblz4-dev

For MPI, you can either install mpich with

sudo apt-get install mpich

or OpenMPI with

sudo apt-get install libopenmpi-dev

Step 3 - Install Blosc

First clone the Blosc Github repo:

git clone https://github.com/Blosc/c-blosc.git

Then run:

cd c-blosc
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX='/usr'
cmake --build .
ctest
sudo cmake --build . -- target install

Step 4 - Install Zstandard

First get and unzip the build folder:

wget https://github.com/facebook/zstd/archive/v1.1.0.tar.gz
tar xf v1.1.0.tar.gz

Then install the library:

cd zstd-1.0.0
sudo make install PREFIX='/usr'

Step 5 - Install Google Test

First run:

sudo apt-get install libgtest-dev

This command does not create the library - you need to do some manual building. Run the following:

cd /usr/src/gtest
sudo cmake .
sudo make
sudo mv libgtest* /usr/lib/

Step 6 - Install Doxygen

If you wish to compile the TileDB code documentation, you need Doxygen. Simply run:

sudo apt-get install doxygen

Step 7 - Clone the TileDB Repo

Run:

git clone https://github.com/Intel-HLS/TileDB.git

This will create a directory TileDB in you current working directory, containing the TileDB source files.

Step 8 - Build TileDB

Now you are ready to build TileDB. First, enter the TileDB source file directory:

cd <TileDB source file directory path>

Then, create a new directory, say build, and enter this directory:

mkdir build
cd build

Next, run

cmake ..

This automatically finds all TileDB dependencies and creates the Makefile. Subsequently, create the TileDB C library:

make -j X

where X is the practically the level of parallelism for make (e.g., we use 4 on our machine).

To install the (static and shared) libraries, run

sudo make install

This typically installs the TileDB libraries in /usr/local/lib and the header files in /usr/local/include.

To create the examples, run:

make examples -j X

To run the code tests, run:

make check -j X

You can create the Doxygen documentation by running:

make doc

Finally, you can clean up the built files by running:

make clean

Please make sure to read the Compilation Flags and Build Troubleshooting sections below for some important information you may need depending on your machine and preferences.

# Build on CentOS Linux

Step 1 - Install cmake, wget and git

You first need to install two useful programs, cmake, GNU wget and git. Simply run:

sudo yum install cmake
sudo yum install wget
sudo yum install git

Step 2 - Install zlib, OpenSSL, LZ4, Blosc and MPI

Simply run:

sudo yum install zlib-devel
sudo yum install openssl-devel
sudo yum install epel-release
sudo yum install lz4-devel
sudo yum install blosc-devel

For MPI, you can either install mpich with

sudo yum install mpich-devel

or OpenMPI with

sudo yum install openmpi-devel

Step 3 - Install Zstandard

First get and unzip the build folder:

wget https://github.com/facebook/zstd/archive/v1.1.0.tar.gz
tar xf v1.1.0.tar.gz

Then install the library:

cd zstd-1.0.0
sudo make install PREFIX='/usr'

Step 4 - Install Google Test

Get the Google Test development release:

wget https://github.com/google/googletest/archive/release-1.7.0.tar.gz
tar xf release-1.7.0.tar.gz

Compile the sources:

cd googletest-release-1.7.0
cmake .
make

Move the header and library files:

sudo mv include/gtest /usr/include/gtest
sudo mv libgtest_main.a libgtest.a /usr/lib/

Step 5 - Install Doxygen

If you wish to compile the TileDB code documentation, you need Doxygen. Simply run:

sudo yum install doxygen

Step 6 - Clone the TileDB Repo

Run:

git clone https://github.com/Intel-HLS/TileDB.git

This will create a directory TileDB in you current working directory, containing the TileDB source files.

Step 7 - Build TileDB

Now you are ready to build TileDB. First, enter the TileDB source file directory:

cd <TileDB source file directory path>

Then, create a new directory, say build, and enter this directory:

mkdir build
cd build

Next, run

cmake ..

This automatically finds all TileDB dependencies and creates the Makefile. Subsequently, create the TileDB C library:

make -j X

where X is the practically the level of parallelism for make (e.g., we use 4 on our machine).

To install the (static and shared) libraries, run

sudo make install

This typically installs the TileDB libraries in /usr/local/lib and the header files in /usr/local/include.

To create the examples, run:

make examples -j X

To run the code tests, run:

make check -j X

You can create the Doxygen documentation by running:

make doc

Finally, you can clean up the built files by running:

make clean

Please make sure to read the Compilation Flags and Build Troubleshooting sections below for some important information you may need depending on your machine and preferences.

# Build on Mac OS X

Mac OS X uses Clang as its default C++ compiler, which unfortunately does not support OpenMP. Therefore, the TileDB builder deactivates OpenMP under Mac OS X, which comes at some small performance cost (e.g., internal sorting is not parallel any more when loading unsorted sparse cells into an array).

Step 1 - Install Homebrew

First, you need to install Homebrew, which is a package manager for Mac OS X. After installing Homebrew, you will be able to install packages to your Mac by running:

brew install <package-name>

You can find the installation guidelines on its website, but an easy way to install Homebrew is by running the following command:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Step 2 - Install cmake, wget and git

You also need to install three useful programs, cmake, GNU wget and git. Simply run

brew install cmake
brew install wget
brew install git

Step 3 - Install zlib, OpenSSL, MPI and LZ4

Simply run

brew install lzlib
brew install openssl
brew install mpich
brew install lz4

Note that we have tested with the mpich version of MPI, but one may alternatively install OpenMPI.

Step 4 - Install Blosc

First clone the Blosc Github repo:

git clone https://github.com/Blosc/c-blosc.git

Then run:

cd c-blosc
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX='/usr'
cmake --build .
ctest
sudo cmake --build . -- target install

Step 5 - Install Zstandard

First get and unzip the build folder:

wget https://github.com/facebook/zstd/archive/v1.1.0.tar.gz
tar xf v1.1.0.tar.gz

Then install the library:

cd zstd-1.0.0
sudo make install PREFIX='/usr'

Step 6 - Install Google Test

Get the Google Test development release:

wget https://github.com/google/googletest/archive/release-1.7.0.tar.gz
tar xf release-1.7.0.tar.gz

Compile the sources:

cd googletest-release-1.7.0
cmake .
make

Move the header and library files:

sudo mv include/gtest /usr/include/gtest
sudo mv libgtest_main.a libgtest.a /usr/lib/

Note: The last commands may not work on your Max OS due to Apple's System Integrity Protection. If that is the case, you need to reboot your system and press cmd+r while it is booting to enter the recovery mode. Then go to Utilities > Terminal from the menu at the top, and run the following commands:

csrutil disable
reboot

This will reboot your system, and then you will be able to move the header and library files as described above.

Step 7 - Install Doxygen

If you wish to compile the TileDB code documentation, you need Doxygen. Simply run:

brew install doxygen

Step 8 - Clone the TileDB Repo

Run:

git clone https://github.com/Intel-HLS/TileDB.git

This will create a directory TileDB in you current working directory, containing the TileDB source files.

Step 9 - Build TileDB

Now you are ready to build TileDB. First, enter the TileDB source file directory:

cd <TileDB source file directory path>

Then, create the TileDB C API library by running

make -j X

where X is the number of jobs to run simultaneously (typically equal to the number of available cores - this will speed up the make process significantly).

To create the examples, run:

make examples -j X

To run the code tests, run:

make test -j X

Note that the compiler may complain about missing OpenSSL header files, or the linker may complain about missing the corresponding library. In this case you may have to run:

make INCLUDE_PATHS=-I<your-openssl-include-path> LIBRARY_PATHS=-L<your-openssl-lib-path> -j X

You can create the Doxygen documentation by running:

make doc

Finally, you can clean up the built files by running:

make clean
# Proper Care for MPI

Assuming that you have installed MPI (mpich in our examples), you need to add some extra flags upon making. Specifically, you need to run:

make MPI=1 CXX=mpic++ MPILIB=-lmpich -j X

Specifically for Mac OS X and depending on its version, if you attempt to run an MPI-based program with mpiexec, the program may fail with a weird message about the hostname. You need to do the following. First, go to the System Preferences > Sharing settings and make sure you enable Remote Login. Then, set a proper host name by running:

sudo scutil HostName your-host-name

where your-host-name is a name of your choice. Next, add 127.0.0.1 your-host-name to the /etc/hosts/ file. Finally, add your public key to the authorized keys file by running:

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

The above should do the trick.

# Compilation Flags

You can pass various flags to the compiler through make when building the TileDB library. Specifically:

  • To control the type of build, set BUILD=debug for debug mode, and BUILD=release for release mode (the default is release mode if you omit this flag).

  • To enable verbose error messages, add flag VERBOSE=1 (the default is 0, i.e., no error messages).

  • To enable OpenMP for multi-threading, add flag OPENMP=1 (the default is 0, i.e., OpenMP is disabled). Note that OpenMP is not supported with Clang on Mac OS X.

  • TileDB needs your MAC address for creating certain folder names (this is important if you use TileDB in a parallel file system). The MAC address is typically retrieved through some network interface, e.g., "eth0" for late Ubuntu versions, and "en0" for Mac OS X. If your network interface is not "eth0" for Ubuntu or "en0" for Mac OS X, you need to find which interface carries your MAC address (by running ifconfig in terminal) and then set it with flag MAC_ADDRESS_INTERFACE, e.g., MAC_ADDRESS_INTERFACE=em0.

  • You can tune the compression level of GZIP, Zstandard and Blosc by setting flags COMPRESSION_LEVEL_GZIP, COMPRESSION_LEVEL_ZSTD and COMPRESSION_LEVEL_BLOSC, respectively. If you omit these flags, TileDB will use their respective defaults ones.

  • You can enable/disable parallel sort in the case of sparse arrays when you write in unsorted mode through flag USE_PARALLE_SORT, e.g., USE_PARALLE_SORT=1 enables the parallel sort. Note that parallel sort is disabled by default.

# Build Troubleshooting # Test If It Works

Try running the tutorial examples. For example,

./examples/bin/release/tiledb_workspace_group_create

should create a directory called my_workspace in your current working directory.

Moreover, you can try running the current Google tests, e.g.,

make test -j 4

which should show that all tests passed.

Table of Contents

Clone this wiki locally