Test all demos on ARM

[fhennig]

We already did some work on https://github.com/stackabletech/demos/tree/spike/demos-on-arm which leaves just the demos open not touched right now.

Shouldn't be to much. The test would be to have a ARM64 cluster and see if the demos work as expected (Be aware that it will only work for 23.11 and stackable-experimental contains ARM64 images and not multi arch for it)

[adwk67]

General comments

• add something to the docs about using mixed-arch clusters (e.g. set nodeSelector for daemonsets, pods, jobs etc.)
• stacklet endpoints do not work for some of the demos

Demos

• ✔️ airflow-scheduled-job: corrected the images: b13ae63, 5f11fba
• ✔️ hbase-hdfs-load-cycling-data
• ✔️ nifi-kafka-druid-earthquake-data: 5f11fba
• ✔️ nifi-kafka-druid-water-level-data: 5f11fba
• ❌ spark-k8s-anomaly-detection-taxi-data: possibly fixable, requires docker.stackable.tech/demos/pyspark-k8s-with-kafka-and-iceberg:3.3.0-stackable0.2.0 as arm64
• ✔️ trino-iceberg: some trino queries work, but connectivity is unstable (maybe cluster-related) e.g.
  • Query 20240308_091808_00028_q7hxj failed: Failed to query OPA backend
  • Query 20240308_090407_00024_q7hxj failed: Failed connecting to Hive metastore
• ✔️ trino-taxi-data: 6a6d783
• ❌ jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data: possibly fixable, requires docker.stackable.tech/demos/pyspark-k8s-with-scikit-learn:3.3.0-stackable23.4 as arm64
• ✔️ logging
• ❌ signal-processing: timescale chart does not seem to be multi-arch and is no longer maintained (may be fixable using a postgres operator instead)
• ❌ question: data-lakehouse-iceberg-trino-spark: possibly fixable, requires docker.stackable.tech/demos/pyspark-k8s-with-kafka-and-iceberg:3.3.0-stackable0.2.0 as arm

[sbernauer]

@adwk67 would you be willing to switch docker.stackable.tech/demos/pyspark-k8s-with-kafka-and-iceberg:3.3.0-stackable0.2.0 to pull the deps using spec.deps.packages similar to the following? This way it should (tm) work (and a thing we want anyway I guess :) )

Using Spark 3.5 something like:

  deps:
    packages:
      - org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.4.3
      - org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.0

[adwk67]

As discussed today, closing this with fix issues carried over into #25 and #26.