[WIP] Deploy the assembly as an artifact

[rmannibucau]

What changes were proposed in this pull request?

Re-enable assembly artifacts - not that I'm not 100% sure of release process so maybe I missed something.
Not sure the link with ./dev/make-distribution.sh

Why are the changes needed?

Being able to download spark distro (ideally any flavor but the base one is the most important) enables to provide custom distro with prepackaged bundles (like Apache Iceberg for ex).
--packages option can be tricky on some env or with some deps.

Does this PR introduce any user-facing change?

No zip published, nothing breaks.

How was this patch tested?

Not tested in release process.

Was this patch authored or co-authored using generative AI tooling?

Nop.

[rmannibucau]

@dongjoon-hyun hello, can you guide me on how you do the releases so I can attach the tgz/zip to the deployment properly please? This would be a great enhancement to the release

[dongjoon-hyun]

As you know, Apache Spark community intentionally dropped assembly way at Apache Spark 2.0.0 due to many issues including security, @rmannibucau .

• #11796

[dongjoon-hyun]

Let me close this because this is a great regression from Apache Spark community perspective. We can continue our discussion on the closed PR.

This would be a great enhancement to the release

[dongjoon-hyun]

Personally, I don't recommend assembly because I believe assembly feature has not been maintained properly since 2.0.0.

[rmannibucau]

@dongjoon-hyun the assembly is what is proposed on apache spark website download so spark didnt drop anything, just dropped the automotion and central publication which is negative from an user standpoint and leads to issues in downstream usages and automotion since the download urls are not stable (it would be from central.

Also note that from a security standpoint it is not worse than all apache spark distro (from the tgz of the download area to the docker image) by design.

So overall I don't see why not fixing the convenient deliverable fro mmy window, will help the community and not hurt spark more than it is today since the bundles are archives automatically anyway and must be "immutable" (in the spirit since nothing is never immutable).

Can you please revise it since it doesn't impact spark project more than having to push the binary(ies) on nexus?

[dongjoon-hyun]

Could you give me the specific link of that part from Apache Spark website?

the assembly is what is proposed on apache spark website

I thought you are trying to build a far jar like Apache Spark 1.6.x. Did I understand your question correctly?

[rmannibucau]

@dongjoon-hyun this is what I'm referring https://spark.apache.org/downloads.html (you know, latest and previous are not using the same link and archives.apache.org are not considered stable so both cases are broken to consume the zip/tgz).

I thought you are trying to build a far jar like Apache Spark 1.6.x. Did I understand your question correctly?

No, I'm more trying to do a custom distro to use on local machine of ops to interact with a Spark Cluster - but I need to add a bunch of jars and props.

[rmannibucau]

@dongjoon-hyun any hope we work on that issue so the assembly is consummable with maven dependency plugin "natively"? Happy to adjust the PR once I know how you do release it (if manual it is fine to keep it closed and just add it in the release steps for me if you do prefer)