BUG: When create a Java data delivery pipeline, a NoClassDefFoundError exception is throw when deploying metadata service

[csun-cpointe]

Description

Following the regression test script to create a java spark data delivery pipeline, when deploying the metadata service, a NoClassDefFoundError exception is shown in the log:

2025-04-21 16:23:59,595 INFO  [io.quarkus] (main) extensions-metadata-service 1.13.0-SNAPSHOT on JVM (powered by Quarkus 3.8.6) started in 3.582s. Listening on: http://0.0.0.0:8080/
2025-04-21 16:23:59,596 INFO  [io.quarkus] (main) Profile prod activated. 
2025-04-21 16:23:59,596 INFO  [io.quarkus] (main) Installed features: [cdi, kafka-client, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-reactive-messaging, smallrye-reactive-messaging-kafka, vertx]
2025-04-21 16:24:07,477 INFO  [io.sma.rea.mes.kafka] (vert.x-eventloop-thread-4) SRMSG18256: Initialize record store for topic-partition 'metadata-ingest-0' at position -1.
2025-04-21 16:24:07,494 ERROR [io.sma.rea.mes.kafka] (vert.x-eventloop-thread-4) SRMSG18203: A message sent to channel `metadata-ingest` has been nacked, fail-stop
2025-04-21 16:24:07,494 WARN  [io.sma.rea.mes.kafka] (vert.x-eventloop-thread-4) SRMSG18228: A failure has been reported for Kafka topics '[metadata-ingest]': java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/analysis/NoSuchTableException
	at com.boozallen.aiops.metadata.hive.HiveMetadataAPIService_Bean.proxy(Unknown Source)
	at com.boozallen.aiops.metadata.hive.HiveMetadataAPIService_Bean.get(Unknown Source)
	at com.boozallen.aiops.metadata.hive.HiveMetadataAPIService_Bean.get(Unknown Source)
	at com.boozallen.aiops.metadata.MetadataIngest_Bean.doCreate(Unknown Source)
	at com.boozallen.aiops.metadata.MetadataIngest_Bean.create(Unknown Source)
	at com.boozallen.aiops.metadata.MetadataIngest_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)

Steps to Reproduce

Clear, specific, and detailed steps taken to enable reproduction of the bug for investigation.

1. Navigate to the regression test script.
2. Follow the steps for Getting Started, Instantiating the Regression Test Project, and Adding Only Spark Data Delivery Pipeline
3. From the command line, run the mvn clean install command and execute the specified manual actions until no more manual actions are required
4. Follow the Deploying the Regression Test Project step to deploy the project
5. After all services are ready, click into the metadata service

Expected Behavior

A clear and concise description of the outcome you expected from the actions taken.
6. The metadata service should run without any errors

Actual Behavior

A clear and concise description of the outcome you experienced from the actions taken.
6. An exception is shown in the log

2025-04-21 16:23:59,595 INFO  [io.quarkus] (main) extensions-metadata-service 1.13.0-SNAPSHOT on JVM (powered by Quarkus 3.8.6) started in 3.582s. Listening on: http://0.0.0.0:8080/
2025-04-21 16:23:59,596 INFO  [io.quarkus] (main) Profile prod activated. 
2025-04-21 16:23:59,596 INFO  [io.quarkus] (main) Installed features: [cdi, kafka-client, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-reactive-messaging, smallrye-reactive-messaging-kafka, vertx]
2025-04-21 16:24:07,477 INFO  [io.sma.rea.mes.kafka] (vert.x-eventloop-thread-4) SRMSG18256: Initialize record store for topic-partition 'metadata-ingest-0' at position -1.
2025-04-21 16:24:07,494 ERROR [io.sma.rea.mes.kafka] (vert.x-eventloop-thread-4) SRMSG18203: A message sent to channel `metadata-ingest` has been nacked, fail-stop
2025-04-21 16:24:07,494 WARN  [io.sma.rea.mes.kafka] (vert.x-eventloop-thread-4) SRMSG18228: A failure has been reported for Kafka topics '[metadata-ingest]': java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/analysis/NoSuchTableException
	at com.boozallen.aiops.metadata.hive.HiveMetadataAPIService_Bean.proxy(Unknown Source)
	at com.boozallen.aiops.metadata.hive.HiveMetadataAPIService_Bean.get(Unknown Source)
	at com.boozallen.aiops.metadata.hive.HiveMetadataAPIService_Bean.get(Unknown Source)
	at com.boozallen.aiops.metadata.MetadataIngest_Bean.doCreate(Unknown Source)
	at com.boozallen.aiops.metadata.MetadataIngest_Bean.create(Unknown Source)
	at com.boozallen.aiops.metadata.MetadataIngest_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.c12(Unknown Source)
	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.computeIfAbsent(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.impl.ClientProxies.getApplicationScopedDelegate(ClientProxies.java:21)
	at com.boozallen.aiops.metadata.MetadataIngest_ClientProxy.arc$delegate(Unknown Source)
	at com.boozallen.aiops.metadata.MetadataIngest_ClientProxy.ingest(Unknown Source)
	at com.boozallen.aiops.metadata.MetadataIngest_SmallRyeMessagingInvoker_ingest_d5e5108c31eb1feb4008d09260281337ac4eebbf.invoke(Unknown Source)
	at io.smallrye.reactive.messaging.providers.AbstractMediator.invoke(AbstractMediator.java:160)
	at io.smallrye.reactive.messaging.providers.AbstractMediator.lambda$invokeOnMessageContext$13(AbstractMediator.java:168)
	at io.smallrye.reactive.messaging.providers.locals.LocalContextMetadata.lambda$invokeOnMessageContext$0(LocalContext

Expected Behavior

A clear and concise description of the outcome you expected from the actions taken.

Actual Behavior

A clear and concise description of the outcome you experienced from the actions taken.

Additional Context

• Log output
• Screenshots (if applicable)
• Solution Baseline Version
• Environment details (local, cloud, Azure, AWS, etc.)

[epwilkins]

The verification for this was done via IT for the aissemble-metadata Docker image.