Custom transformation for Cassandra to Spanner SourceDB #2201

taherkl · 2025-02-18T06:36:13Z

This pull request introduces the following key enhancements:

Enhanced Null Handling:

Improved handling of null values for all column types, including String columns, ensuring consistent data integrity and preventing unintended null conversions.
This enhancement covers all possible column data types, providing a more robust and error-free data transformation process.

Custom Transformation Logic:

Added support for custom transformation logic specifically for Spanner to Cassandra Reverse migrations.
This includes the ability to handle complex data type mappings and custom transformations needed for seamless data synchronization between Cassandra and Spanner.

* sync upstream/main (#98) * Custom transformation fixes * Added Custom Transformation * Added Custom Transformation * Added Fixes * Address null to all columns * Added Null Assert * Added Timeout fixes * Added Spotless fixes * reverse merge the main * Added Custom Fixes * Added Drop Keys ---------

Move null test case into previous tests

Referesh `SpannerToSourceDbCustomTransformationIT` tables for re-runs…

shreyakhajanchi · 2025-02-18T14:36:07Z

...rcedb/src/test/java/com/google/cloud/teleport/v2/templates/SpannerToCassandraSourceDbIT.java

@@ -78,6 +79,7 @@ public class SpannerToCassandraSourceDbIT extends SpannerToSourceDbITBase {
  private static final String USER_TABLE = "Users";
  private static final String USER_TABLE_2 = "Users2";
  private static final String ALL_DATA_TYPES_TABLE = "AllDatatypeColumns";
+  private static final String ALL_DATA_TYPES_TABLE_FOR_NULL_KEY = "AllDataTypeColumnsForNullKey";


not required?

@shreyakhajanchi went unknowingly, I have removed the same will able to reflect soon

shreyakhajanchi · 2025-02-18T14:36:46Z

...rcedb/src/test/java/com/google/cloud/teleport/v2/templates/SpannerToCassandraSourceDbIT.java

+        Mutation.newInsertOrUpdateBuilder(ALL_DATA_TYPES_TABLE)
+            .set("varchar_column")
+            .to("SampleVarcharForNull") // Only this column has a value
+            .set("tinyint_column")


why set null for others, we can simply skip setting other columns and it will be NULL automatically, right?

@shreyakhajanchi as per our discussion we are going to add another case where we are going to add only value for the primary key alone

shreyakhajanchi · 2025-02-18T14:37:00Z

...rcedb/src/test/java/com/google/cloud/teleport/v2/templates/SpannerToCassandraSourceDbIT.java

@@ -538,8 +631,8 @@ private void assertAllDataTypeRowsInCassandraDB()
    PipelineOperator.Result result =
        pipelineOperator()
            .waitForCondition(
-                createConfig(jobInfo, Duration.ofMinutes(10)),
-                () -> getRowCount(ALL_DATA_TYPES_TABLE) == 1);
+                createConfig(jobInfo, Duration.ofMinutes(20)),


Is it taking more than 10 min?

No @shreyakhajanchi Will reduce to 10

shreyakhajanchi · 2025-02-18T14:39:14Z

...o-sourcedb/src/test/resources/SpannerToCassandraSourceIT/cassandra-transformation-schema.sql

+    "first_name" text,
+    "last_name" text


why are these two columns in quotes?

Yes we can remove

shreyakhajanchi · 2025-02-18T14:40:30Z

...-to-sourcedb/src/test/resources/SpannerToCassandraSourceIT/spanner-transformation-schema.sql

+DROP TABLE IF EXISTS customers;
+CREATE TABLE IF NOT EXISTS customers (
+    id INT64 NOT NULL,
+    full_name STRING(125),


full_name column not required in spanner schema?

Yes @shreyakhajanchi as we are trying to populate full name in Cassandra based on FirstName and lastName

shreyakhajanchi · 2025-02-18T14:52:20Z

...rcedb/src/test/java/com/google/cloud/teleport/v2/templates/SpannerToCassandraSourceDbIT.java

@@ -210,7 +212,8 @@ private void assertDeleteRowInCassandraDB() throws InterruptedException {
    PipelineOperator.Result result =
        pipelineOperator()
            .waitForCondition(
-                createConfig(jobInfo, Duration.ofMinutes(10)), () -> getRowCount(USER_TABLE) == 0);
+                createConfig(jobInfo, Duration.ofMinutes(10)),
+                () -> getRowCount(USER_TABLE_2) == 0);


I guess this should still be USER_TABLE and and the table in below should change because the writes to users2 is with txn tag and is filtered:

private void writeDeleteInSpanner() { Mutation insertOrUpdateMutation = Mutation.newInsertOrUpdateBuilder(USER_TABLE_2) .set("id") .to(4) .set("full_name") .to("GG") .build(); spannerResourceManager.write(insertOrUpdateMutation); KeySet allRows = KeySet.all(); Mutation deleteAllMutation = Mutation.delete(USER_TABLE_2, allRows); spannerResourceManager.write(deleteAllMutation); }

@shreyakhajanchi Address the same

shreyakhajanchi · 2025-02-18T14:53:57Z

...com/google/cloud/teleport/v2/templates/SpannerToCassandraSourceDBCustomTransformationIT.java

+    spannerResourceManager.write(m1);
+  }
+
+  /**


Remove these elaborate comments, the methods are self explainatory

Sure @shreyakhajanchi

shreyakhajanchi · 2025-02-18T15:04:12Z

...panner-custom-shard/src/main/java/com/custom/CustomTransformationWithCassandraForLiveIT.java

+      if (request.getEventType().equals("UPDATE")) {
+        return new MigrationTransformationResponse(null, false);
+      }
+      if (request.getEventType().equals("DELETE")) {
+        return new MigrationTransformationResponse(null, true);
+      }


not required

@shreyakhajanchi as per current logic in IT we are populating full name based on first name and last name so I guess it is need to be here

shreyakhajanchi · 2025-02-18T15:04:27Z

...panner-custom-shard/src/main/java/com/custom/CustomTransformationWithCassandraForLiveIT.java

+    if (request.getTableName().equals("Customers")) {
+      Map<String, Object> row = new HashMap<>(request.getRequestRow());
+      row.put("full_name", row.get("first_name") + " " + row.get("last_name"));
+      row.put("migration_shard_id", request.getShardId() + "_" + row.get("id"));
+      MigrationTransformationResponse response = new MigrationTransformationResponse(row, false);
+      return response;
+    }


not required

@shreyakhajanchi as per current logic in IT we are populating full name based on first name and last name so I guess it is need to be here

shreyakhajanchi · 2025-02-18T15:04:34Z

...panner-custom-shard/src/main/java/com/custom/CustomTransformationWithCassandraForLiveIT.java

@@ -0,0 +1,70 @@
+/*
+ * Copyright (C) 2024 Google LLC


@shreyakhajanchi address the same

codecov · 2025-02-19T07:39:02Z

Codecov Report

Attention: Patch coverage is 50.00000% with 26 lines in your changes missing coverage. Please review.

Project coverage is 47.05%. Comparing base (810e553) to head (a8008cf).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...om/CustomTransformationWithCassandraForLiveIT.java	0.00%	22 Missing ⚠️
...2/templates/dbutils/dml/CassandraDMLGenerator.java	92.30%	0 Missing and 2 partials ⚠️
...v2/templates/dbutils/dml/CassandraTypeHandler.java	50.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2201      +/-   ##
============================================
- Coverage     47.06%   47.05%   -0.01%     
- Complexity     4048     4378     +330     
============================================
  Files           876      877       +1     
  Lines         52223    52270      +47     
  Branches       5505     5515      +10     
============================================
+ Hits          24577    24594      +17     
- Misses        25885    25911      +26     
- Partials       1761     1765       +4

Components	Coverage Δ
spanner-templates	`68.83% <50.00%> (-0.08%)`	⬇️
spanner-import-export	`65.64% <ø> (-0.04%)`	⬇️
spanner-live-forward-migration	`76.48% <ø> (ø)`
spanner-live-reverse-replication	`78.49% <50.00%> (-0.31%)`	⬇️
spanner-bulk-migration	`87.94% <ø> (ø)`

Files with missing lines	Coverage Δ
...2/templates/dbutils/dml/CassandraDMLGenerator.java	`91.15% <92.30%> (+0.75%)`	⬆️
...v2/templates/dbutils/dml/CassandraTypeHandler.java	`83.58% <50.00%> (-0.77%)`	⬇️
...om/CustomTransformationWithCassandraForLiveIT.java	`0.00% <0.00%> (ø)`

... and 3 files with indirect coverage changes

pull-request-size bot added the size/XL label Feb 18, 2025

Move null test case into previous tests

e9f0beb

Move null test case into previous tests

pull-request-size bot added size/XXL and removed size/XL labels Feb 18, 2025

taherkl and others added 2 commits February 18, 2025 13:16

Merge pull request #109 from GoogleCloudPlatform/main

5b599a4

Referesh `SpannerToSourceDbCustomTransformationIT` tables for re-runs…

Issue fix

df151c9

pull-request-size bot added size/XL and removed size/XXL labels Feb 18, 2025

pawankashyapollion added 2 commits February 18, 2025 16:48

Null fix

3f7e766

Null fix for varint

e615cd8

taherkl marked this pull request as ready for review February 18, 2025 12:35

taherkl requested a review from a team as a code owner February 18, 2025 12:35

taherkl requested review from asthamohta and bharadwaj-aditya February 18, 2025 12:35

Removed unwanted code

a8008cf

shreyakhajanchi reviewed Feb 18, 2025

View reviewed changes

pawankashyapollion added 3 commits February 19, 2025 20:45

Added Bytes Fixes

27c006a

Missing Bytes Error Code Pushed

ff076e9

PR comment fixes (#114)

9f46ed8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom transformation for Cassandra to Spanner SourceDB #2201

Custom transformation for Cassandra to Spanner SourceDB #2201

taherkl commented Feb 18, 2025 •

edited

Loading

shreyakhajanchi Feb 18, 2025

pawankashyapollion Feb 20, 2025

shreyakhajanchi Feb 18, 2025

pawankashyapollion Feb 20, 2025

shreyakhajanchi Feb 18, 2025

pawankashyapollion Feb 20, 2025

shreyakhajanchi Feb 18, 2025

pawankashyapollion Feb 20, 2025

shreyakhajanchi Feb 18, 2025

pawankashyapollion Feb 20, 2025

shreyakhajanchi Feb 18, 2025

pawankashyapollion Feb 20, 2025

shreyakhajanchi Feb 18, 2025

pawankashyapollion Feb 20, 2025

shreyakhajanchi Feb 18, 2025

pawankashyapollion Feb 20, 2025

shreyakhajanchi Feb 18, 2025

pawankashyapollion Feb 20, 2025

shreyakhajanchi Feb 18, 2025

pawankashyapollion Feb 20, 2025

codecov bot commented Feb 19, 2025 •

edited

Loading

Custom transformation for Cassandra to Spanner SourceDB #2201

Are you sure you want to change the base?

Custom transformation for Cassandra to Spanner SourceDB #2201

Conversation

taherkl commented Feb 18, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Feb 19, 2025 • edited Loading

Codecov Report

taherkl commented Feb 18, 2025 •

edited

Loading

codecov bot commented Feb 19, 2025 •

edited

Loading