[SPARK-53658][CONNECT] Accept literal data_type field in ExecutePlanResponse

[heyihong]

What changes were proposed in this pull request?

This PR enable the Spark Connect to support accepting literal data_type fields in ExecutePlanResponse messages. The main purpose is to let the server know that the client can accept the data_type field in Literal messages, ensuring compatibility with older clients that don't support this new field.

Key Changes:

1. Protocol Buffer Updates:

   • Added new AcceptResponseOptions message in base.proto with accept_literal_data_type_field boolean field
   • Updated ExecutePlanRequest to include accept_response_options field

2. Client-Side Implementation:

   • Added ACCEPT_RESPONSE_OPTIONS constant in SparkConnectClient.scala
   • Integrated response options into request building process

3. Server-Side Processing:

   • Enhanced ExecuteHolder.scala with acceptLiteralDataTypeField lazy val to check client capabilities
   • Updated SparkConnectPlanExecution.scala to accept and utilize the new parameter
   • Modified ExecuteThreadRunner.scala to pass the new parameter through the execution chain

Why are the changes needed?

The current Spark Connect implementation needs to handle compatibility between new and old clients regarding the data_type field in Literal messages. This enhancement is needed to:

1. Backward Compatibility: Ensure older clients that don't support the new data_type field in Literal messages continue to work without issues
2. Server Awareness: Allow the server to know which clients can accept the data_type field, enabling appropriate response formatting
3. Graceful Migration: Provide a smooth transition path for clients to adopt the new data type field support

Does this PR introduce any user-facing change?

No

How was this patch tested?

build/sbt "connect-client-jvm/testOnly *ClientE2ETestSuite -- -z SPARK-53490"
build/sbt "connect/testOnly *SparkConnectServiceSuite"

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor 1.6.42