[Do not merge] Validate Beam 2.63.0rc2 #2177

Abacn · 2025-02-12T16:04:34Z

No description provided.

Abacn · 2025-02-12T20:45:52Z

org.tensorflow.proto does not support protobuf 4:

The latest release of

https://mvnrepository.com/artifact/org.tensorflow/proto

was in 2019.

This prevents classic Templates to use protobuf 4, that is now used by Beam 2.63.0

Abacn · 2025-02-12T23:29:39Z

sourcedb-to-spanner tests timing out

codecov · 2025-02-12T23:41:06Z

Codecov Report

Attention: Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.

Project coverage is 46.99%. Comparing base (b93cad8) to head (aaab86d).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...apache/beam/sdk/io/gcp/spanner/LocalSpannerIO.java	0.00%	4 Missing ⚠️
...streamstopubsub/BigtableChangeStreamsToPubSub.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2177      +/-   ##
============================================
+ Coverage     46.97%   46.99%   +0.02%     
+ Complexity     4049     4046       -3     
============================================
  Files           876      874       -2     
  Lines         52193    52161      -32     
  Branches       5502     5498       -4     
============================================
- Hits          24517    24514       -3     
+ Misses        25920    25895      -25     
+ Partials       1756     1752       -4

Components	Coverage Δ
spanner-templates	`68.91% <ø> (-0.01%)`	⬇️
spanner-import-export	`65.72% <ø> (-0.02%)`	⬇️
spanner-live-forward-migration	`76.50% <ø> (ø)`
spanner-live-reverse-replication	`78.77% <ø> (ø)`
spanner-bulk-migration	`87.94% <ø> (ø)`

Files with missing lines	Coverage Δ
...e/cloud/teleport/templates/BigQueryToTFRecord.java	`19.69% <ø> (ø)`
...treamstopubsub/model/TestChangeStreamMutation.java	`0.00% <ø> (ø)`
...streamstopubsub/BigtableChangeStreamsToPubSub.java	`0.00% <0.00%> (ø)`
...apache/beam/sdk/io/gcp/spanner/LocalSpannerIO.java	`0.00% <0.00%> (ø)`

... and 3 files with indirect coverage changes

Abacn · 2025-02-13T15:36:40Z

all tests currently failing are PythonUDF tests.

Downloaded "surefire-integration-test-results.zip" manually, decompress and run grep -B 1 -nr 'error message' shows all test failures are PythonUDF tests, which are expected.

./python/target/surefire-reports/TEST-com.google.cloud.teleport.templates.python.PubSubToBigQueryYamlIT.xml-79-  <testcase name="testPubSubToBigQuery" classname="com.google.cloud.teleport.templates.python.PubSubToBigQueryYamlIT" time="964.957">
./python/target/surefire-reports/TEST-com.google.cloud.teleport.templates.python.PubSubToBigQueryYamlIT.xml:80:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_18_00_21-17100005783908586268?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_18_00_21-17100005783908586268?project=cloud-teleport-testing.
--
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.TextIOToBigQueryIT.xml-80-  <testcase name="testTextIOToBigQueryWithPythonUdf" classname="com.google.cloud.teleport.v2.templates.TextIOToBigQueryIT" time="1,532.708">
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.TextIOToBigQueryIT.xml:81:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_11-17125510468052756728?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_11-17125510468052756728?project=cloud-teleport-testing.
--
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToBigQueryIT.xml-82-  <testcase name="testPubsubToBigQueryWithPythonUdf" classname="com.google.cloud.teleport.v2.templates.PubSubToBigQueryIT" time="1,417.968">
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToBigQueryIT.xml:83:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_31-6459326777086093214?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_31-6459326777086093214?project=cloud-teleport-testing.
--
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToBigQueryAutoIT.xml-84-  <testcase name="testPubsubToBigQueryWithPythonUdf" classname="com.google.cloud.teleport.v2.templates.PubSubToBigQueryAutoIT" time="1,538.468">
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToBigQueryAutoIT.xml:85:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_31-14586942559005192948?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_31-14586942559005192948?project=cloud-teleport-testing.
--
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.TextToBigQueryStreamingIT.xml-79-  <testcase name="testTextToBigQueryWithPythonUDFs" classname="com.google.cloud.teleport.v2.templates.TextToBigQueryStreamingIT" time="1,536.09">
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.TextToBigQueryStreamingIT.xml:80:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_30-16711561795242901948?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_30-16711561795242901948?project=cloud-teleport-testing.
--
./v2/googlecloud-to-elasticsearch/target/surefire-reports/TEST-com.google.cloud.teleport.v2.elasticsearch.templates.BigQueryToElasticsearchIT.xml-81-  <testcase name="testBigQueryToElasticsearchWithPythonUdf" classname="com.google.cloud.teleport.v2.elasticsearch.templates.BigQueryToElasticsearchIT" time="1,104.803">
./v2/googlecloud-to-elasticsearch/target/surefire-reports/TEST-com.google.cloud.teleport.v2.elasticsearch.templates.BigQueryToElasticsearchIT.xml:82:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_30_14-16069209254424499837?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_30_14-16069209254424499837?project=cloud-teleport-testing.
--
./v2/googlecloud-to-elasticsearch/target/surefire-reports/TEST-com.google.cloud.teleport.v2.elasticsearch.templates.GCSToElasticsearchIT.xml-83-  <testcase name="testElasticsearchCsvWithoutHeadersPython" classname="com.google.cloud.teleport.v2.elasticsearch.templates.GCSToElasticsearchIT" time="1,378.181">
./v2/googlecloud-to-elasticsearch/target/surefire-reports/TEST-com.google.cloud.teleport.v2.elasticsearch.templates.GCSToElasticsearchIT.xml:84:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_32_47-15516695606867294538?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_32_47-15516695606867294538?project=cloud-teleport-testing.
--
./v2/googlecloud-to-elasticsearch/target/surefire-reports/TEST-com.google.cloud.teleport.v2.elasticsearch.templates.PubSubToElasticsearchIT.xml-79-  <testcase name="testPubSubToElasticsearchWithPythonUdf" classname="com.google.cloud.teleport.v2.elasticsearch.templates.PubSubToElasticsearchIT" time="1,273.772">
./v2/googlecloud-to-elasticsearch/target/surefire-reports/TEST-com.google.cloud.teleport.v2.elasticsearch.templates.PubSubToElasticsearchIT.xml:80:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_32_03-1910802050061065714?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_32_03-1910802050061065714?project=cloud-teleport-testing.
--
./v2/pubsub-binary-to-bigquery/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubsubProtoToBigQueryIT.xml-79-  <testcase name="pubSubProtoToBigQueryWithPythonUdf" classname="com.google.cloud.teleport.v2.templates.PubsubProtoToBigQueryIT" time="1,186.835">
./v2/pubsub-binary-to-bigquery/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubsubProtoToBigQueryIT.xml:80:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_18_35_16-18294704747315617489?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_18_35_16-18294704747315617489?project=cloud-teleport-testing.
--
./v2/pubsub-to-mongodb/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToMongoDBIT.xml-79-  <testcase name="testPythonPubsubToMongoDB" classname="com.google.cloud.teleport.v2.templates.PubSubToMongoDBIT" time="1,104.131">
./v2/pubsub-to-mongodb/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToMongoDBIT.xml:80:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_18_56_06-2339488772302340482?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_18_56_06-2339488772302340482?project=cloud-teleport-testing.

Abacn · 2025-02-13T16:42:34Z

Kafka PR: https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/13312637711

Spanner PR: https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/13312639690

Abacn · 2025-02-13T17:57:26Z

Single test failed:

2025-02-13T17:20:34.0661569Z [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1,543.391 s <<< FAILURE! - in com.google.cloud.teleport.v2.templates.SpannerToSourceDbCustomTransformationIT
2025-02-13T17:20:34.0663779Z [ERROR] testCustomTransformation(com.google.cloud.teleport.v2.templates.SpannerToSourceDbCustomTransformationIT)  Time elapsed: 1,543.391 s  <<< FAILURE!
2025-02-13T17:20:34.0665084Z value of  : result.check if meets all conditions
2025-02-13T17:20:34.0665627Z expected  : CONDITION_MET
2025-02-13T17:20:34.0666021Z but was   : TIMEOUT
2025-02-13T17:20:34.0666472Z result was: TIMEOUT
2025-02-13T17:20:34.0669433Z 	at com.google.cloud.teleport.v2.templates.SpannerToSourceDbCustomTransformationIT.assertRowInMySQL(SpannerToSourceDbCustomTransformationIT.java:350)
2025-02-13T17:20:34.0672234Z 	at com.google.cloud.teleport.v2.templates.SpannerToSourceDbCustomTransformationIT.testCustomTransformation(SpannerToSourceDbCustomTransformationIT.java:166)

test infra issue, other spanner-to-sourcedb IT passed, ignore for now

Abacn · 2025-02-13T18:04:53Z

I see, test failure is actually relevant:

java.lang.NoClassDefFoundError: Could not initialize class com.google.spanner.v1.Session$LabelsDefaultEntryHolder
2025-02-13 12:35:35.242 EST
at com.google.spanner.v1.Session.internalGetLabels(Session.java:147)
2025-02-13 12:35:35.242 EST
at com.google.spanner.v1.Session.getSerializedSize(Session.java:490)
2025-02-13 12:35:35.242 EST
at com.google.protobuf.CodedOutputStream.computeMessageSizeNoTag(CodedOutputStream.java:861)
2025-02-13 12:35:35.242 EST
at com.google.protobuf.CodedOutputStream.computeMessageSize(CodedOutputStream.java:641)
2025-02-13 12:35:35.242 EST
at com.google.spanner.v1.BatchCreateSessionsRequest.getSerializedSize(BatchCreateSessionsRequest.java:232)
2025-02-13 12:35:35.242 EST
at io.grpc.protobuf.lite.ProtoInputStream.available(ProtoInputStream.java:108)
2025-02-13 12:35:35.242 EST
at io.grpc.internal.MessageFramer.getKnownLength(MessageFramer.java:204)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.ExceptionInInitializerError [in thread "grpc-default-executor-3"]
2025-02-13 12:35:35.242 EST
at com.google.spanner.v1.Session$LabelsDefaultEntryHolder.<clinit>(Session.java:132)
2025-02-13 12:35:35.242 EST
... 17 more

Abacn · 2025-02-13T19:49:39Z

Spanner PR:

First run:

     🧪 - MySQLSourceDbToSpannerSimpleIT | The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-13_09_34_20-15299568926532995869?project=cloud-teleport-testing.
     🧪 - SpannerToSourceDbCustomTransformationIT | value of  : result.check if meets all conditions

Second run:

🧪 - TimezoneIT | The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-13_10_19_21-17776592963417510888?project=cloud-teleport-testing.
     🧪 - PostgreSQLDataTypesIT | The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-13_10_51_31-2701783821855436363?project=cloud-teleport-testing.
     🧪 - SpannerChangeStreamToGcsSingleShardIT | The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-13_10_17_33-14641481078103719277?project=cloud-teleport-testing.

example error:

2025-02-13 13:21:47.309 EST
Caused by: com.google.cloud.spanner.SpannerException: RESOURCE_EXHAUSTED: Timed out after waiting 60000ms for acquiring session. To mitigate error SessionPoolOptions#setAcquireSessionTimeout(Duration) to set a higher timeout or increase the number of sessions in the session pool.

different tests failed in two runs. proceed.

Kafka PR: passed

Abacn · 2025-02-18T20:32:48Z

For who going to upgrading Templates: remember to remove https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/2177/files#diff-21f0f2b4485d5b8359efec95a27f2bd574a0e46885f815cab18cc8eb847b9b69L706 (fixed by #2182) and re-generate requirements for Python templates

pull-request-size bot added the size/L label Feb 12, 2025

Abacn added ignore-for-release and removed size/L labels Feb 12, 2025

Abacn added 2 commits February 12, 2025 16:36

[Do not merge] Validate Beam 2.63.0rc1

2643e14

fixes

171b958

Abacn force-pushed the validate-beam-263rc1 branch from 1ee90bf to 171b958 Compare February 12, 2025 18:38

pull-request-size bot added the size/XL label Feb 12, 2025

spotless apply

6610531

Abacn added 2 commits February 12, 2025 21:32

Fix tests

118846c

Update tensorflow class

0f12564

Remove dependency pin managed by beam-gcp-bom

5c9bc02

Abacn changed the title ~~[Do not merge] Validate Beam 2.63.0rc1~~ [Do not merge] Validate Beam 2.63.0rc2 Feb 13, 2025

validate rc2

aaab86d

Abacn mentioned this pull request Feb 13, 2025

[Bug]: Flaky Spanner Integration test due to "Could not initialize class com.google.spanner.v1.Session$LabelsDefault" #2191

Open

Abacn mentioned this pull request Feb 20, 2025

Upgrade Beam version to 2.63.0 #2206

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Do not merge] Validate Beam 2.63.0rc2 #2177

[Do not merge] Validate Beam 2.63.0rc2 #2177

Abacn commented Feb 12, 2025

Abacn commented Feb 12, 2025

Abacn commented Feb 12, 2025

codecov bot commented Feb 12, 2025 •

edited

Loading

Abacn commented Feb 13, 2025

Abacn commented Feb 13, 2025 •

edited

Loading

Abacn commented Feb 13, 2025

Abacn commented Feb 13, 2025

Abacn commented Feb 13, 2025

Abacn commented Feb 18, 2025

[Do not merge] Validate Beam 2.63.0rc2 #2177

Are you sure you want to change the base?

[Do not merge] Validate Beam 2.63.0rc2 #2177

Conversation

Abacn commented Feb 12, 2025

Abacn commented Feb 12, 2025

Abacn commented Feb 12, 2025

codecov bot commented Feb 12, 2025 • edited Loading

Codecov Report

Abacn commented Feb 13, 2025

Abacn commented Feb 13, 2025 • edited Loading

Abacn commented Feb 13, 2025

Abacn commented Feb 13, 2025

Abacn commented Feb 13, 2025

Abacn commented Feb 18, 2025

codecov bot commented Feb 12, 2025 •

edited

Loading

Abacn commented Feb 13, 2025 •

edited

Loading