BUG: Spark Operator does not properly use Shared Ivy cache

[epwilkins]

Description

The shared Ivy Cache, as described by the docs, allows Spark Operator and all Spark applications to share a single cache. However, Spark Operator does not use this cache and so libraries are always downloaded twice on the first run of an application: once for Spark Operator, and once for the application. Subsequent runs of the application re-use the cache and the operator already has the file. However, re-deploying the application will cause Spark Operator to need to download the libraries again (though the application still uses the cache).

It looks like this is caused by the mount path of the PVC. The chart uses /home/spark/.ivy but the user's HOME path is /opt/spark.

Steps to Reproduce

1. Create a downstream project
2. Add a spark pipeline (ensure packages has entries in the application values files)
3. Deploy the application
4. Execute the Spark pipeline

Expected Behavior

The Spark pipeline driver logs indicate the packages were already retrieved.

Actual Behavior

The Spark pipeline driver logs indicate the packages had to be downloaded

Additional Context

aiSSEMBLE version: 2.0.0