feat(spanner): Support dynamic certificate and key rotation in Spanner Omni

[sagnghos]

Summary

This PR adds support for zero-downtime dynamic reloading of client certificates/keys (mTLS) and server root CA certificates in Spanner Omni without requiring application or connection pool restarts.

Changes

• DynamicKeyManager (com.google.cloud.spanner.omni): An X509ExtendedKeyManager that monitors file modification timestamps and lengths, dynamically reloading rotated client certificates and RSA/EC private keys.
• DynamicTrustManager (com.google.cloud.spanner.omni): An X509ExtendedTrustManager that dynamically reloads updated server root CA certificates into an in-memory keystore/trust manager upon file changes.
• SpannerOptions & Connection API:
  • Added Builder.setCaCertificate(String caCertificate) and getCaCertificate() across SpannerOptions, ConnectionProperties, ConnectionOptions, and SpannerPool.
  • Updated Builder.useClientCert(String, String) to use dynamic key management.
• SpannerOmniHelper: Added support for spanner.ca_cert_path and updated mTLS setup detection when client certificates are provided.
• Testing: Added unit tests covering dynamic certificate/key rotation, CA rotation, multi-CA bundles, fallback handling, and options configuration.

Fixes b/562755231

[gemini-code-assist]

Code Review

This pull request introduces dynamic loading and automatic reloading of client certificates, private keys, and server root CA certificates for Spanner Omni instances by implementing DynamicKeyManager and DynamicTrustManager. It also updates SpannerOptions, ConnectionOptions, and related classes to support the new caCertificate configuration. Feedback on these changes highlights critical performance concerns regarding blocking file I/O operations performed on every TLS handshake or trust check, which could block Netty's EventLoop threads; throttling these checks is recommended. Additionally, it is advised to remove the direct dependency on BouncyCastle in DynamicKeyManager to prevent classpath conflicts, relying instead on standard Java APIs for PKCS#8 private keys.

[sagnghos]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for configuring a server root CA certificate (caCertificate) for SSL/TLS authentication in Spanner Omni instances, alongside new DynamicKeyManager and DynamicTrustManager classes that dynamically reload certificates and keys from disk upon rotation. Feedback on these changes suggests enhancing private key parsing in DynamicKeyManager to explicitly reject PKCS#1 keys with a clear error message, and ensuring that temporary self-signed certificates created during testing in SpannerOptionsTest are properly cleaned up in a try-finally block to prevent file leaks.

[gemini-code-assist]

The current implementation of parsePrivateKey only supports PKCS#8 private keys (which start with -----BEGIN PRIVATE KEY-----). If a user provides a PKCS#1 private key (which starts with -----BEGIN RSA PRIVATE KEY----- or -----BEGIN EC PRIVATE KEY-----), the parsing will fail with a cryptic InvalidKeySpecException because PKCS8EncodedKeySpec cannot parse PKCS#1 structures.

Detecting PKCS#1 keys and throwing a clear, actionable IllegalArgumentException with instructions on how to convert the key to PKCS#8 using openssl will significantly improve the user experience. Additionally, using StandardCharsets.UTF_8 is more robust than StandardCharsets.US_ASCII for general PEM files that might contain UTF-8 comments or metadata.

  private static PrivateKey parsePrivateKey(byte[] keyBytes) throws Exception {
    String keyStr = new String(keyBytes, StandardCharsets.UTF_8);
    if (keyStr.contains("-----BEGIN RSA PRIVATE KEY-----") || keyStr.contains("-----BEGIN EC PRIVATE KEY-----")) {
      throw new IllegalArgumentException(
          "PKCS#1 private keys are not supported. Please convert your key to PKCS#8 format using: "
              + "openssl pkcs8 -topk8 -nocrypt -in <key> -out <key_pkcs8>");
    }
    byte[] der;
    if (keyStr.contains("-----BEGIN PRIVATE KEY-----")) {
      der = extractPemContent(keyStr, "-----BEGIN PRIVATE KEY-----", "-----END PRIVATE KEY-----");
    } else {
      try {
        der = Base64.getMimeDecoder().decode(keyBytes);
      } catch (IllegalArgumentException e) {
        der = keyBytes;
      }
    }

    PKCS8EncodedKeySpec spec = new PKCS8EncodedKeySpec(der);
    try {
      return KeyFactory.getInstance("RSA").generatePrivate(spec);
    } catch (Exception e) {
      return KeyFactory.getInstance("EC").generatePrivate(spec);
    }
  }

[gemini-code-assist]

The SelfSignedCertificate class from Netty creates temporary files in the system's default temporary directory. If these certificates are not explicitly deleted by calling delete(), they will leak and clutter the system's temp directory over time, especially in CI/CD environments where tests are run frequently.

Wrapping the test logic in a try-finally block and calling delete() on both ssc and ca ensures that these temporary files are always cleaned up properly.

    io.grpc.netty.shaded.io.netty.handler.ssl.util.SelfSignedCertificate ssc =
        new io.grpc.netty.shaded.io.netty.handler.ssl.util.SelfSignedCertificate("spanner.test");
    io.grpc.netty.shaded.io.netty.handler.ssl.util.SelfSignedCertificate ca =
        new io.grpc.netty.shaded.io.netty.handler.ssl.util.SelfSignedCertificate("spanner.ca");

    try {
      String certPath = ssc.certificate().getAbsolutePath();
      String keyPath = ssc.privateKey().getAbsolutePath();
      String caPath = ca.certificate().getAbsolutePath();

      SpannerOptions options =
          SpannerOptions.newBuilder()
              .setProjectId("test-project")
              .setCredentials(NoCredentials.getInstance())
              .setHost("https://localhost:1234")
              .useClientCert(certPath, keyPath)
              .setCaCertificate(caPath)
              .build();

      assertNotNull(options.getChannelConfigurator());

      SpannerOptions fromBuilder = options.toBuilder().build();
      assertNotNull(fromBuilder.getChannelConfigurator());

      // Test standalone setCaCertificate
      SpannerOptions caOnlyOptions =
          SpannerOptions.newBuilder()
              .setProjectId("test-project")
              .setCredentials(NoCredentials.getInstance())
              .setHost("https://localhost:1234")
              .setCaCertificate(caPath)
              .build();

      assertNotNull(caOnlyOptions.getChannelConfigurator());

      // Test setCaCertificate combined with login (username/password)
      SpannerOptions loginWithCaOptions =
          SpannerOptions.newBuilder()
              .setProjectId("test-project")
              .setType(SpannerOptions.InstanceType.OMNI)
              .setHost("https://localhost:1234")
              .setCaCertificate(caPath)
              .login("test-user", "test-pass".toCharArray())
              .build();

      assertTrue(loginWithCaOptions.getCredentials() instanceof SpannerOmniCredentials);
      assertNotNull(loginWithCaOptions.getChannelConfigurator());
    } finally {
      ssc.delete();
      ca.delete();
    }