Load Tests - Cassandra Reverse Replication #2163

taherkl · 2025-01-31T12:43:51Z

This PR introduces Load Testing for the Spanner to Cassandra DB pipeline to evaluate performance, and reliability under high data loads.

Key Changes:

Added Load Test Scenarios
- Simulates high-throughput data migration from Spanner to Cassandra
- Tests different batch sizes and concurrent transactions
- Measures latency, throughput, and error rates
Implemented Performance Metrics Logging
- Tracks execution time, failed vs. successful transactions
- Captures Cassandra write latencies and Spanner read times

Sync master

Sync main branch

tests: Adding Forward Migration Tests (GoogleCloudPlatform#2001)

Sync main branch

Use [self-hosted, it] for prepare java cache workflow (GoogleCloudPlatform#2080)

Sync main branch

Metadata config and pipeline options (GoogleCloudPlatform#2081)

fix: eager source row fetching logic (GoogleCloudPlatform#2071)

sync upstream/main

Sync main with upstream

Sync main branch

Sync main

Support avro arrays for postgres insertion. (GoogleCloudPlatform#2154)

* Addition of Load Tests in SpannerToSourceDB For Cassandra

codecov · 2025-02-05T09:58:03Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 46.90%. Comparing base (2631a1d) to head (f9b1e3e).
Report is 10 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #2163   +/-   ##
=========================================
  Coverage     46.89%   46.90%           
- Complexity     4346     4351    +5     
=========================================
  Files           874      874           
  Lines         52070    52090   +20     
  Branches       5461     5468    +7     
=========================================
+ Hits          24420    24433   +13     
- Misses        25906    25909    +3     
- Partials       1744     1748    +4

Components	Coverage Δ
spanner-templates	`68.82% <ø> (+0.01%)`	⬆️
spanner-import-export	`65.70% <ø> (+0.05%)`	⬆️
spanner-live-forward-migration	`76.50% <ø> (ø)`
spanner-live-reverse-replication	`78.67% <ø> (ø)`
spanner-bulk-migration	`87.87% <ø> (ø)`

see 2 files with indirect coverage changes

...-sourcedb/src/test/java/com/google/cloud/teleport/v2/templates/SpannerToCassandraLTBase.java

...ner-to-sourcedb/src/test/java/com/google/cloud/teleport/v2/templates/CassandraRowsCheck.java

darshan-sj · 2025-02-19T05:14:20Z

it/cassandra/src/main/java/org/apache/beam/it/cassandra/conditions/CassandraRowsCheck.java

+      throw new IllegalArgumentException("CassandraResourceManager must not be null.");
+    }
+
+    // Use reflection to call executeStatement if it exists


Why is reflection needed? Can you give some context?

@darshan-sj as resourceManager object's type (T) is generic, and the method executeStatement is not part of a well-defined interface or superclass that resourceManager is guaranteed to implement. Reflection allows here to dynamically invoke the executeStatement method on the resourceManager object if available

darshan-sj · 2025-02-19T05:24:30Z

...-sourcedb/src/test/java/com/google/cloud/teleport/v2/templates/CassandraResourceManager.java

+   */
+  public synchronized ResultSet executeStatement(String statement) {
+    LOG.info("Executing statement: {}", statement);
+    return this.executeStatement(statement, DEFAULT_CASSANDRA_TIMEOUT);


We are introducing a 2 second timeout on this default executeStatement method. We should not modify the behavior for other tests which are using this method.

@darshan-sj I guess request timeout is 2 sec only so does not required any changes for other method

darshan-sj · 2025-02-19T05:26:12Z

...-sourcedb/src/test/java/com/google/cloud/teleport/v2/templates/SpannerToCassandraLTBase.java

+import org.apache.beam.it.gcp.storage.GcsResourceManager;
+
+/**
+ * Base class for Spanner to sourcedb Load tests. It provides helper functions related to


Correct the comment also to reflect Spanner to Cassandra Load tests.

Addressed @darshan-sj

darshan-sj · 2025-02-19T05:27:59Z

...-sourcedb/src/test/java/com/google/cloud/teleport/v2/templates/SpannerToCassandraLTBase.java

+  private static final String TEMPLATE_SPEC_PATH =
+      MoreObjects.firstNonNull(
+          TestProperties.specPath(),
+          "gs://dataflow-templates-spanner-to-cassandra/templates/flex/Spanner_to_SourceDb");


Where is this bucket gs://dataflow-templates-spanner-to-cassandra/ ?

Addressed @darshan-sj

darshan-sj · 2025-02-19T05:38:40Z

...-sourcedb/src/test/java/com/google/cloud/teleport/v2/templates/SpannerToCassandraLTBase.java

+      MoreObjects.firstNonNull(
+          TestProperties.specPath(),
+          "gs://dataflow-templates-spanner-to-cassandra/templates/flex/Spanner_to_SourceDb");
+  public CassandraResourceManager cassandraSharedResourceManager;


Why is this variable called `SharedResourceManager? How is it shared?

Addressed @darshan-sj it is typo mistake

darshan-sj · 2025-02-19T05:43:42Z

...o-sourcedb/src/test/java/com/google/cloud/teleport/v2/templates/SpannerToSourceDbLTBase.java

-                getGcsPath(artifactBucket, "input/shard.json", gcsResourceManager));
+                getGcsPath(
+                    artifactBucket,
+                    !Objects.equals(sourceType, MYSQL_SOURCE_TYPE)


Please use template parameters overriding instead of having a ternary operator here.
Example:

DataflowTemplates/v2/datastream-to-spanner/src/test/java/com/google/cloud/teleport/v2/templates/loadtesting/DataStreamToSpannerLTBase.java

Line 152 in b51c933

params.putAll(templateParameters);

@darshan-sj address the same

Sync main branch

taherkl and others added 19 commits November 27, 2024 20:19

Merge remote-tracking branch 'upstream/main'

15b621d

Merge pull request #9 from GoogleCloudPlatform/main

d314c70

Sync master

Merge pull request #20 from GoogleCloudPlatform/main

71a6477

Sync main branch

Merge pull request #24 from GoogleCloudPlatform/main

7512825

Sync main branch

Merge pull request #26 from GoogleCloudPlatform/main

5aa21dc

tests: Adding Forward Migration Tests (GoogleCloudPlatform#2001)

Merge pull request #28 from GoogleCloudPlatform/main

04bac39

Sync main branch

Merge pull request #29 from GoogleCloudPlatform/main

f618128

Use [self-hosted, it] for prepare java cache workflow (GoogleCloudPlatform#2080)

Merge pull request #32 from GoogleCloudPlatform/main

e53e8e4

Sync main branch

Merge pull request #36 from GoogleCloudPlatform/main

407a593

Sync main branch

Merge pull request #40 from GoogleCloudPlatform/main

0d140e6

Sync main branch

Merge pull request #46 from GoogleCloudPlatform/main

ae3037a

Sync main branch

Merge pull request #48 from GoogleCloudPlatform/main

ef5ae8d

Metadata config and pipeline options (GoogleCloudPlatform#2081)

Merge pull request #52 from GoogleCloudPlatform/main

c6442ba

fix: eager source row fetching logic (GoogleCloudPlatform#2071)

Merge pull request #55 from GoogleCloudPlatform/main

7c02a96

sync upstream/main

Merge pull request #79 from GoogleCloudPlatform/main

e3f4369

Sync main with upstream

Merge pull request #83 from GoogleCloudPlatform/main

5d5f1a8

Sync main branch

Merge pull request #84 from GoogleCloudPlatform/main

f1a4d1e

Sync main

Merge pull request #87 from GoogleCloudPlatform/main

22e677f

Support avro arrays for postgres insertion. (GoogleCloudPlatform#2154)

* Addition of Load Tests in SpannerToSourceDB For Cassandra (#89)

43b6699

* Addition of Load Tests in SpannerToSourceDB For Cassandra

pull-request-size bot added the size/XL label Jan 31, 2025

Address Merge conflict

f4c9811

taherkl marked this pull request as ready for review February 5, 2025 07:07

taherkl requested a review from a team as a code owner February 5, 2025 07:07

taherkl requested review from Deep1998 and asthamohta February 5, 2025 07:07

pawankashyapollion added 2 commits February 5, 2025 13:52

Added LT Refectored (#92)

30f1c7c

Added POM Dependecies

f9b1e3e

shreyakhajanchi reviewed Feb 11, 2025

View reviewed changes

...-sourcedb/src/test/java/com/google/cloud/teleport/v2/templates/SpannerToCassandraLTBase.java Show resolved Hide resolved

...ner-to-sourcedb/src/test/java/com/google/cloud/teleport/v2/templates/CassandraRowsCheck.java Outdated Show resolved Hide resolved

shreyakhajanchi requested a review from darshan-sj February 11, 2025 06:17

shreyakhajanchi added the ignore-for-release label Feb 11, 2025

pawankashyapollion added 3 commits February 12, 2025 12:09

Merge branch 'main' into cassandra_rr_load_tests

2528917

RR LOAD TEST FIXES (#101)

c319cbe

Merge branch 'main' into cassandra_rr_load_tests

72d4807

darshan-sj reviewed Feb 19, 2025

View reviewed changes

taherkl and others added 2 commits February 19, 2025 12:31

Merge pull request #113 from GoogleCloudPlatform/main

60c33a1

Sync main branch

Resolved PR comments (#115)

a5a73c2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Load Tests - Cassandra Reverse Replication #2163

Load Tests - Cassandra Reverse Replication #2163

taherkl commented Jan 31, 2025 •

edited

Loading

codecov bot commented Feb 5, 2025 •

edited

Loading

darshan-sj Feb 19, 2025

pawankashyapollion Feb 20, 2025

darshan-sj Feb 19, 2025

pawankashyapollion Feb 20, 2025

darshan-sj Feb 19, 2025

pawankashyapollion Feb 20, 2025

darshan-sj Feb 19, 2025

pawankashyapollion Feb 20, 2025

darshan-sj Feb 19, 2025

pawankashyapollion Feb 20, 2025

darshan-sj Feb 19, 2025

pawankashyapollion Feb 20, 2025

Load Tests - Cassandra Reverse Replication #2163

Are you sure you want to change the base?

Load Tests - Cassandra Reverse Replication #2163

Conversation

taherkl commented Jan 31, 2025 • edited Loading

codecov bot commented Feb 5, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

taherkl commented Jan 31, 2025 •

edited

Loading

codecov bot commented Feb 5, 2025 •

edited

Loading