[jiangxt2]

[#13489] feat(spark-connector): Add the governed Doris batch-read baseline

What changes were proposed in this pull request?

This pull request adds a governed Apache Doris batch-read baseline for the Gravitino Spark connector.

• Add the explicit spark.sql.gravitino.enableDorisSupport opt-in for Spark 3.5 / Scala 2.12.
• Preserve the existing generic JDBC path when the opt-in is disabled or when another JDBC provider is used.
• Add a specialized read-only Doris catalog facade that requires vended JDBC credentials and catalog-managed JDBC URL and driver properties.
• Preserve existing spark.bypass.* catalog properties and ordinary Spark JDBC read options. Match application overrides ignoring key case while preserving native JDBC property spelling, such as connectTimeout and sslMode, when forwarding to the driver.
• Reject protected connection, credential, table/query, and Spark initialization-SQL options in application catalog/scan options and catalog spark.bypass.* properties, including the driver options dbname and createDatabaseIfNotExist.
• Validate ordinary Doris tables against FE SHOW COLUMNS metadata and JDBC metadata, selecting only the requested database and exact table and checking column order, names, ordinary type signatures, and nullability.
• Use the Spark table's JDBCOptions and dialect connection factory for physical schema validation, preserving the configured driver and connection properties, including SSL settings and connection timeouts.
• Accept the confirmed SMALLINT representation difference only when FE and JDBC report SMALLINT, the logical Spark type is ShortType, and Spark JDBC metadata uses IntegerType. Keep ShortType scans through the existing JDBC table and converter, with all other schema checks retained.
• Redact failures from the first Spark JDBC physical-schema load and Doris metadata validation: omit raw messages, causes, and suppressed exceptions, and retain the failure stage and available safe SQLState/vendor diagnostics. Missing-driver ClassNotFoundException initialization failures remain diagnosable.
• Fail closed for unsupported types and metadata mismatches, and reject Spark writes and catalog DDL through the specialized Doris path.
• Add documentation, focused unit tests, and Doris integration tests for the read contract.

Column names/order, FE/JDBC signatures, and nullability remain validated. CHAR and FLOAT are rejected because their Gravitino logical types differ from the types exposed by Spark JDBC.

The existing Spark JDBC V2 delegate determines supported pushdowns and partitioned reads. Doris-specific optimizations and routing for aggregates, Top-N, limits, offsets, and partitioned reads, plus native tablet reads, special-type normalization, and write support, are outside this PR.

Why are the changes needed?

A Doris-specific entry point is needed to require vended credentials, validate the physical schema, and expose read-only capabilities. Keeping this path opt-in preserves the existing JDBC behavior by default. Physical schema checks need the table's effective Spark JDBC configuration, and existing catalog bypass settings need to remain effective in the specialized path.

Native Connector/J property names must retain their exact spelling for timeout and TLS settings to remain effective. The first Spark JDBC schema load precedes Doris validation and also needs redaction because Spark can include the full JDBC URL in connection errors. Rejecting known database-target and automatic-creation options helps contain connection side effects.

In Spark 3.5.3, JDBC metadata represents SMALLINT as IntegerType, while the existing Gravitino JDBC table scans it as ShortType using getShort(). Recognizing this specific difference avoids rejecting valid SMALLINT columns and preserves their Spark type. The rule is confined to the Doris validator and does not change shared method/interface contracts.

The specialized path uses MySQL Connector/J to access the Doris FE MySQL protocol. The driver and other required runtime dependencies remain external to the connector artifact and must be available on the Spark driver and executors.

Fix: #13489

Part of: #12846

Does this PR introduce any user-facing change?

Yes.

• Adds the spark.sql.gravitino.enableDorisSupport Spark configuration key. Its default is false.
• When enabled for a jdbc-doris catalog, the connector selects the governed read-only Doris path.
• The specialized path requires a vended JDBC credential and catalog-managed JDBC URL and driver properties.
• Existing spark.bypass.* catalog properties supply ordinary JDBC defaults; application options take precedence when keys match ignoring case. Spark read-option names are case-insensitive, but native Connector/J properties must use exact driver spelling, such as connectTimeout and sslMode.
• Application catalog/scan options and catalog spark.bypass.* properties cannot override the connection URL, driver, credentials, or table/query target, or supply Spark's prepareQuery/sessionInitStatement or the driver's dbname/createDatabaseIfNotExist.
• SMALLINT remains Spark ShortType, matching generic reads of the same Doris table.
• Doris DATETIME is exposed as Spark TimestampType; string rendering follows spark.sql.session.timeZone.
• Writes and Spark catalog DDL are rejected for the specialized Doris path.
• Generic JDBC behavior for other providers and for jdbc-doris without the opt-in remains unchanged.

Read-only capabilities reject Spark writes and catalog DDL; they do not guarantee that arbitrary JDBC driver configuration is free of connection or session side effects. Vended database credentials should have only the required read and metadata permissions.

Applications must keep org.apache.spark.sql.execution.datasources.jdbc.connection logging at INFO or higher and disable JDBC driver connection/protocol tracing on both Spark drivers and executors. Adapter redaction covers physical-schema loading and validation errors; it does not sanitize arbitrary third-party logs or executor exception chains. The Doris documentation includes a Log4j 2 configuration example.

How was this patch tested?

Local validation used Corretto 17.0.18, Gradle 8.2, Spark 3.5.3 / Scala 2.12, MySQL Connector/J 8.0.33, and Doris 3.0.6.2 Docker fixtures. Gravitino ran in embedded test mode.

• Module Spotless, RAT, and git diff --check passed.
• The four affected suites, TestDorisPhysicalSchemaValidator35, TestDorisPropertiesConverter35, TestDorisReadOnlyTable35, and TestGravitinoDorisCatalogSpark35, passed 26 unit tests, with zero failures, errors, or skips.
• Unit coverage includes schema/SMALLINT guards, actual driver.connect(Properties) inputs, native property spelling and override precedence, protected options at application/catalog-bypass/scan boundaries, missing credentials and driver classes, first-load redaction (URL, nested cause, and suppressed details), authorization before physical access, read-only/DDL rejection, resource closure, and safe diagnostics. TLS checks cover property forwarding and precedence only.
• Generic/specialized registration and fallback checks also passed in TestGravitinoDriverPlugin.
• The complete SparkJdbcDorisReadBaselineIT35 suite passed 22 Doris-backed tests, with zero failures, errors, or skips: 15 parameterized scalar spelling cases, one SMALLINT generic/specialized comparison, and six other read-contract regression tests.
• All 16 allowlisted type spellings are covered: BIGINT, BOOLEAN, DATE, DATEV2, plain DATETIME, DATETIMEV2(0), DECIMAL(10,2), DECIMALV3(20,4), DOUBLE, INT, INTEGER, SMALLINT, STRING, TEXT, TINYINT, and VARCHAR(64). Each has NOT NULL and nullable columns, non-null values, and actual NULL values. Aliases may resolve to the same FE/JDBC metadata type.
• The SMALLINT comparison reads the same Doris table through generic and specialized catalogs, checking both catalog classes, Spark ShortType, Java Short values, the signed bounds -32768 and 32767, column nullability, and actual NULL values.
• Other IT checks cover specialized catalog registration, scalar query results, exact database/table metadata selection despite _ pattern matches and same-named tables in another database, write-aware load rejection, and unsupported JSON rejection. The bypass-option test verifies correct results, zero pushed predicates, and a retained Spark residual filter when spark.bypass.pushDownPredicate=false. The suite also runs with a catalog-level native connectTimeout default and an application override.

The Doris suite is tagged doris-multi-version and was run locally with -PdorisMultiVersionTest; default Spark CI excludes that tag.

Validation commands:

git diff --check

./gradlew :spark-connector:spark-3.5:spotlessCheck rat

./gradlew :spark-connector:spark-3.5:test \
  --tests org.apache.gravitino.spark.connector.jdbc.doris.TestDorisPhysicalSchemaValidator35 \
  --tests org.apache.gravitino.spark.connector.jdbc.doris.TestDorisPropertiesConverter35 \
  --tests org.apache.gravitino.spark.connector.jdbc.doris.TestDorisReadOnlyTable35 \
  --tests org.apache.gravitino.spark.connector.jdbc.doris.TestGravitinoDorisCatalogSpark35 \
  -PskipITs

./gradlew :spark-connector:spark-3.5:test \
  --tests org.apache.gravitino.spark.connector.plugin.TestGravitinoDriverPlugin \
  -PskipITs

./gradlew :spark-connector:spark-3.5:test \
  --tests org.apache.gravitino.spark.connector.integration.test.jdbc.SparkJdbcDorisReadBaselineIT35 \
  -PskipDockerTests=false -PtestMode=embedded -PdorisMultiVersionTest

[github-actions]

Code Coverage Report

Overall Project	72.15% -0.15%	🟢
Files changed	63.71%	🟢

Module	Coverage
aliyun	19.74%	🔴
api	55.09%	🟢
authorization-common	85.96%	🟢
authorization-ranger	14.44%	🔴
aws	53.54% +14.67%	🟢
aws-bundle	0.0%	🔴
azure	32.1% -14.75%	🔴
azure-bundle	0.0%	🔴
catalog-common	29.49%	🔴
catalog-fileset	82.37%	🟢
catalog-glue	73.24%	🟢
catalog-hive	84.5%	🟢
catalog-jdbc-common	57.98%	🟢
catalog-jdbc-doris	87.56%	🟢
catalog-jdbc-mysql	81.91%	🟢
catalog-jdbc-postgresql	84.02% +4.96%	🟢
catalog-jdbc-starrocks	79.44%	🟢
catalog-kafka	75.21%	🟢
catalog-lakehouse-generic	86.91%	🟢
catalog-lakehouse-hudi	79.1%	🟢
catalog-lakehouse-iceberg	86.8%	🟢
catalog-lakehouse-paimon	84.36%	🟢
catalog-model	77.99%	🟢
cli	44.62%	🟢
client-java	79.44%	🟢
client-java-runtime	0.0%	🔴
common	60.9%	🟢
core	87.5%	🟢
filesystem-hadoop3	76.89%	🟢
flink	55.44%	🟢
flink-common	32.2%	🔴
flink-runtime	0.0%	🔴
gcp	32.2%	🔴
hadoop-auth	68.0%	🟢
hadoop-common	17.84%	🔴
hive-metastore-common	62.3%	🟢
hive-metastore3-libs	58.8%	🟢
iceberg-aliyun-bundle	0.0%	🔴
iceberg-common	67.01%	🟢
iceberg-rest-server	77.74%	🟢
idp-basic	86.77% +1.3%	🟢
integration-test-common	0.0%	🔴
jobs	71.35%	🟢
k8s-job-executor	97.81%	🟢
lance-common	40.49%	🟢
lance-rest-server	69.51%	🟢
lineage	59.39%	🟢
optimizer	84.2%	🟢
optimizer-api	46.58%	🟢
server	91.43%	🟢
server-common	83.89%	🟢
spark	62.67% +0.69%	🟢
tencent	81.78%	🟢
tms	80.0%	🟢
trino-connector	64.88%	🟢

Files

Module	File	Coverage
aws	AwsIrsaCredentialGenerator.java	78.11%	🟢
azure	ADLSTokenGenerator.java	0.0%	🔴
catalog-jdbc-postgresql	PostgreSqlTableOperations.java	87.13%	🟢
idp-basic	IdpCredentialValidator.java	100.0%	🟢
	UpdateUserRequest.java	100.0%	🟢
	IdpRESTUtils.java	95.65%	🟢
	IdpUserGroupManager.java	93.33%	🟢
	IdpUserOperations.java	83.33%	🟢
	IdpAuthorizationFilter.java	75.0%	🟢
spark	DorisPropertiesConverter35.java	93.75%	🟢
	GravitinoSparkPlugin.java	76.47%	🟢
	DorisPhysicalSchemaValidator35.java	72.73%	🟢
	GravitinoDorisCatalogSpark35.java	68.42%	🟢
	DorisReadOnlyTable35.java	66.67%	🟢
spark-common	SparkCatalogKind.java	100.0%	🟢
	SparkBindings.java	100.0%	🟢
	GravitinoDriverPlugin.java	67.93%	🟢
	BaseCatalog.java	24.48%	🔴
	GravitinoSparkConfig.java	0.0%	🔴

[jiangxt2]

This PR introduces a governed, read-only JDBC baseline for Apache Doris. It reuses the existing BaseCatalog SELECT_TABLE authorization path. Required vended credentials and protected-option checks apply only to jdbc-doris, preserving other JDBC providers’ behavior. Type normalization, native tablet reads, writes, and Docker infrastructure changes are outside this PR.

The session-scoped opt-in selects one integration mode for all jdbc-doris catalogs in a Spark application. I would welcome maintainer guidance on whether per-catalog selection is required for this contribution.

[yuqi1129]

Every non-mismatch failure is rethrown without its cause, so users can't tell what went wrong.

A missing driver on the classpath (the most likely setup error, since the docs ask users to provide Connector/J themselves), an unreachable FE, an authentication failure, a vended account without SHOW COLUMNS access, or an unsupported type from typeConverter.toSparkType all end up as the same Doris physical schema validation failed for <ident>. Because this path fails closed, that is the error users will see most often.

Could you:

• keep the cause: new IllegalArgumentException("Doris physical schema validation failed for " + identifier, e). Connector/J messages don't include the password;
• use a private exception subclass for schema mismatches instead of matching the "Doris schema mismatch" message prefix;
• add a unit test with a wrong driver class name that asserts getCause() is a ClassNotFoundException?

[jiangxt2]

Schema checks now run outside the metadata-access try/catch, removing message-prefix matching without adding an exception subclass. Physical-access failures report their stage and available safe SQLState/vendor code. I kept raw causes omitted because Spark 3.5.3’s connection factory can include the JDBC URL in an exception, while #13489 requires connection details to stay out of errors and logs.

Driver loading remains delegated to Spark during catalog initialization. The new regression test enters through GravitinoDorisCatalogSpark35.initialize() with a nonexistent driver class and asserts the directly propagated ClassNotFoundException, the missing class name, and the absence of the JDBC URL and URL/vended password sentinels throughout the printed exception stack. All three catalog unit tests pass.

Would you accept this exception boundary and redaction approach?

[yuqi1129]

ORDINARY_TYPES lists 17 types, but SparkJdbcDorisReadBaselineIT35 only creates INT and VARCHAR(64) columns.

validateColumns requires the FE SHOW COLUMNS signature and the JDBC getColumns signature to be equal. When the two report a type differently, the table is rejected as a schema mismatch and can't be read. I haven't verified these against Doris, but from the code a few look risky:

• datetime with scale 0: FE may print datetime, while JDBC DECIMAL_DIGITS = 0 becomes datetime(0);
• boolean: over the MySQL protocol, JDBC may report TINYINT;
• string / text: JDBC may report a sized VARCHAR, while FE prints string;
• float, double, char, date and decimal aren't exercised either.

Could you add a table to the IT with one column per ORDINARY_TYPES entry, both nullable and NOT NULL, and read the values back? Any type that doesn't match should be normalized in typeSignature or removed from ORDINARY_TYPES and the docs.

[jiangxt2]

The IT now covers all 16 current ORDINARY_TYPES spellings: 15 parameterized cases plus a dedicated SMALLINT comparison. Each spelling has NOT NULL and nullable columns, non-null values, and actual NULL values in the nullable column. Aliases may resolve to the same physical metadata type.

Plain DATETIME is normalized to precision 0, and DECIMALV3 uses the normalized DECIMAL signature. CHAR and FLOAT remain rejected and documented. The narrowly scoped SMALLINT compatibility rule retains Spark ShortType; its dedicated test compares generic and specialized reads of the same Doris table, including signed bounds, Java Short values, and nullability.

Local validation against Doris 3.0.6.2 passed all 22 IT cases. The affected schema-validator suite passed all 14 unit tests, with zero failures, errors, or skips.

[jiangxt2]

@yuqi1129 @diqiu50, could you please review this PR when you have a chance?