D3A Hackathon: Unified Data Analytics with Apache Wayang

Apache Wayang (Incubating) is the first open-source framework that unifies data systems alleviating users from the need of system selection. It decouples applications from platforms so that they can run analytics on one or more platforms seamlessly and efficiently.

Clone repository

Clone this repository in your preferred location using this command:

git clone https://github.com/juripetersen/d3a-hackathon-wayang.git

Go to the root directory:

cd d3a-hackathon-wayang

Docker

For ease of use, we recommend installing Docker, as it allows to pull a Docker Container that comes with all of Apache Wayangs required dependencies. Additionally, we utilize Docker Compose to start our container with all options in one command. We provide a pre-built docker image that contains the necessary tooling in order to run or develop Apache Wayang. The tools necessary for this are:

• Java 11
• Apache Spark
• Hadoop
• Maven

Make sure that docker is up and running.

Step 1: Building the container

In order to create the container, we need to build and start it:

docker compose up -d --build

Note: In case of Apple Silicon M1 Chips, please edit the docker-compose.yaml file and change the image type to apache/incubator-wayang:0.7.1-arm64

Step 2: Connecting to the app container

In order to get an interactive bash session that allows running commands inside of the app container, run the following:

docker exec -it apache-wayang-app bash

Install

Within the root directory (/var/www/html in our container), run the following command to install all required packages, including Wayang:

mvn clean install

Examples

To run any of the following example applications, use this format:

mvn exec:java -Dexec.mainClass="<mainClass>" -Dexec.args="arg1 arg2 ..."

Replace <mainClass> with the main class prefixed with the package it belongs to and <arg1> <arg2> ... with the application-specific parameters that you want to use.

Tasks

We prepared tasks that can be found in TASKS.md