Skip to content
This repository was archived by the owner on Feb 16, 2024. It is now read-only.

Commit dfba191

Browse files
committed
Add demo nifi-kafka-druid-water-level-data (#112)
## Description To run ``` $ git checkout demo-nifi-kafka-druid-water-level-data $ stackablectl --additional-demos-file demos/demos-v1.yaml demo install nifi-kafka-druid-water-level-data ```
1 parent 8c8c046 commit dfba191

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

60 files changed

+4584
-207
lines changed

demos/demos-v1.yaml

Lines changed: 40 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -1,34 +1,23 @@
11
---
22
demos:
3-
trino-taxi-data:
4-
description: Demo loading 2.5 years of New York taxi data into S3 bucket, creating a Trino table and a Superset dashboard
5-
documentation: https://docs.stackable.tech/stackablectl/stable/demos/trino-taxi-data.html
6-
stackableStack: trino-superset-s3
3+
airflow-scheduled-job:
4+
description: Activate a simple Airflow DAG to run continuously at a set interval
5+
stackableStack: airflow
76
labels:
8-
- trino
9-
- superset
10-
- minio
11-
- s3
12-
- ny-taxi-data
7+
- airflow
8+
- job-scheduling
139
manifests:
14-
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/trino-taxi-data/load-test-data.yaml
15-
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/trino-taxi-data/create-table-in-trino.yaml
16-
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/trino-taxi-data/setup-superset.yaml
17-
kafka-druid-water-level-data:
18-
description: Demo ingesting water level data into Kafka, streaming it into Druid and creating a Superset dashboard
19-
documentation: https://docs.stackable.tech/stackablectl/stable/demos/kafka-druid-water-level-data.html
20-
stackableStack: kafka-druid-superset-s3
10+
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/airflow-scheduled-job/enable-and-run-dag.yaml
11+
hbase-hdfs-load-cycling-data:
12+
description: Copy data from S3 bucket to an HBase table
13+
stackableStack: hdfs-hbase
2114
labels:
22-
- kafka
23-
- druid
24-
- superset
25-
- minio
26-
- s3
27-
- water-levels
15+
- hbase
16+
- hdfs
17+
- cycling-tripdata
2818
manifests:
29-
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/kafka-druid-water-level-data/ingest-test-data.yaml
30-
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/kafka-druid-water-level-data/create-druid-ingestion-job.yaml
31-
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/kafka-druid-water-level-data/setup-superset.yaml
19+
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/hbase-hdfs-load-cycling-data/01-distcp-cycling-data.yaml
20+
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/hbase-hdfs-load-cycling-data/02-create-hfile-and-import-to-hbase.yaml
3221
nifi-kafka-druid-earthquake-data:
3322
description: Demo ingesting earthquake data into Kafka using NiFi, streaming it into Druid and creating a Superset dashboard
3423
documentation: https://docs.stackable.tech/stackablectl/stable/demos/nifi-kafka-druid-earthquake-data.html
@@ -45,21 +34,33 @@ demos:
4534
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/nifi-kafka-druid-earthquake-data/create-nifi-ingestion-job.yaml
4635
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/nifi-kafka-druid-earthquake-data/create-druid-ingestion-job.yaml
4736
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/nifi-kafka-druid-earthquake-data/setup-superset.yaml
48-
hbase-hdfs-load-cycling-data:
49-
description: Copy data from S3 bucket to an HBase table
50-
stackableStack: hdfs-hbase
37+
nifi-kafka-druid-water-level-data:
38+
description: Demo ingesting water level data into Kafka using NiFi, streaming it into Druid and creating a Superset dashboard
39+
documentation: https://docs.stackable.tech/stackablectl/stable/demos/nifi-kafka-druid-water-level-data.html
40+
stackableStack: nifi-kafka-druid-superset-s3
5141
labels:
52-
- hbase
53-
- hdfs
54-
- cycling-tripdata
42+
- nifi
43+
- kafka
44+
- druid
45+
- superset
46+
- minio
47+
- s3
48+
- water-levels
5549
manifests:
56-
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/hbase-hdfs-load-cycling-data/01-distcp-cycling-data.yaml
57-
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/hbase-hdfs-load-cycling-data/02-create-hfile-and-import-to-hbase.yaml
58-
airflow-scheduled-job:
59-
description: Activate a simple Airflow DAG to run continously at a set interval
60-
stackableStack: airflow
50+
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/nifi-kafka-druid-water-level-data/create-nifi-ingestion-job.yaml # TODO
51+
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/nifi-kafka-druid-water-level-data/create-druid-ingestion-job.yaml # TODO
52+
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/nifi-kafka-druid-water-level-data/setup-superset.yaml # TODO
53+
trino-taxi-data:
54+
description: Demo loading 2.5 years of New York taxi data into S3 bucket, creating a Trino table and a Superset dashboard
55+
documentation: https://docs.stackable.tech/stackablectl/stable/demos/trino-taxi-data.html
56+
stackableStack: trino-superset-s3
6157
labels:
62-
- airflow
63-
- job-scheduling
58+
- trino
59+
- superset
60+
- minio
61+
- s3
62+
- ny-taxi-data
6463
manifests:
65-
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/airflow-scheduled-job/enable-and-run-dag.yaml
64+
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/trino-taxi-data/load-test-data.yaml
65+
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/trino-taxi-data/create-table-in-trino.yaml
66+
- plainYaml: https://raw.githubusercontent.com/stackabletech/stackablectl/main/demos/trino-taxi-data/setup-superset.yaml

demos/hbase-hdfs-load-cycling-data/01-distcp-cycling-data.yaml

Lines changed: 8 additions & 7 deletions
Large diffs are not rendered by default.

demos/hbase-hdfs-load-cycling-data/02-create-hfile-and-import-to-hbase.yaml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@ metadata:
66
spec:
77
template:
88
spec:
9-
restartPolicy: OnFailure
109
containers:
1110
- name: create-hfile-and-import-to-hbase
1211
image: docker.stackable.tech/stackable/hbase:2.4.12-stackable0.2.0
@@ -43,3 +42,5 @@ spec:
4342
- name: config-volume-hdfs
4443
configMap:
4544
name: hdfs
45+
restartPolicy: OnFailure
46+
backoffLimit: 50

demos/kafka-druid-water-level-data/ingest-test-data.yaml

Lines changed: 0 additions & 96 deletions
This file was deleted.

demos/nifi-kafka-druid-earthquake-data/create-druid-ingestion-job.yaml

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,12 @@ spec:
1313
volumeMounts:
1414
- name: ingestion-job-spec
1515
mountPath: /tmp/ingestion-job-spec
16-
restartPolicy: OnFailure
1716
volumes:
1817
- name: ingestion-job-spec
1918
configMap:
2019
name: create-druid-ingestion-job-spec
21-
restartPolicy: Never
22-
backoffLimit: 50 # It can take some time until Druid is ready
20+
restartPolicy: OnFailure
21+
backoffLimit: 50
2322
---
2423
apiVersion: v1
2524
kind: ConfigMap

demos/nifi-kafka-druid-earthquake-data/create-nifi-ingestion-job.yaml

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,13 +18,12 @@ spec:
1818
valueFrom:
1919
fieldRef:
2020
fieldPath: metadata.namespace
21-
restartPolicy: OnFailure
2221
volumes:
2322
- name: script
2423
configMap:
2524
name: create-nifi-ingestion-job-script
26-
restartPolicy: Never
27-
backoffLimit: 50 # It can take some time until NiFI is ready
25+
restartPolicy: OnFailure
26+
backoffLimit: 50
2827
---
2928
apiVersion: v1
3029
kind: ConfigMap

demos/nifi-kafka-druid-earthquake-data/setup-superset.yaml

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,12 @@ spec:
1313
volumeMounts:
1414
- name: script
1515
mountPath: /tmp/script
16-
restartPolicy: OnFailure
1716
volumes:
1817
- name: script
1918
configMap:
2019
name: setup-superset-script
21-
restartPolicy: Never
22-
backoffLimit: 50 # It can take some time until Superset is ready
20+
restartPolicy: OnFailure
21+
backoffLimit: 50
2322
---
2423
apiVersion: v1
2524
kind: ConfigMap

0 commit comments

Comments
 (0)