[pull] master from apache:master - #498
Merged
Merged
Conversation
… copy (#38983) * [GCP] Optimize BigQuery TriggerCopyJobs performance for WRITE_APPEND * Optimize BigQuery copy jobs in file loads using multi-source copy Updates BigQuery file loads in Python SDK to use multi-source copy jobs when copying temporary tables to the final destination table. * Update BigQueryWrapper._insert_copy_job to support a list of source tables, utilizing BigQuery's multi-source copy capability. * Update TriggerCopyJobs to process temporary tables in batch, splitting them into chunks of 1,200 (BigQuery limit) and triggering multi-source copy jobs. * Implement inline wait for the first chunk in TriggerCopyJobs when write disposition is WRITE_TRUNCATE or WRITE_EMPTY and there are multiple chunks. This ensures the destination table is initialized by the first job before subsequent chunks append to it. * Fix grouping key in _load_data for WRITE_TRUNCATE/WRITE_EMPTY to use the full hashable destination instead of just tableId, preventing incorrect grouping of tables with the same name in different datasets. * Fix TriggerLoadJobs to use bq_wrapper with mock client in tests, resolving credential refresh warnings. * Fix PartitionFiles to avoid yielding empty partitions when a file exceeds limits. TAG=agy CONV=126370d2-f42e-4132-a237-16bd5ccf72a3
* turn off gcp otel auth extension by default.
Fix Flink XVR Tests for Flink 2
Disable flaky SDF finalization tests
Fix cut_release_branch workflow
github-actions
Bot
added
go
java
python
typescript
io
gcp
runners
dataflow
build
website
flink
labels
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )