[pull] master from apache:master - #571
Merged
Merged
Conversation
…ence (#39547) registry.get_coder receives typing and native generic annotations such as tuple[str, float] unconverted and falls back to pickling. Watch now converts hints with convert_to_beam_type before the registry lookup, so annotated poll functions and key functions infer real structured coders and fully typed elements.
…e unnecessary CVE hotfix (#39530) * Bump lower and upper bounds for pyarrow, remove unnecessary CVE hotfix * tighten bounds further, match with our testing environment * transitive deps * Update sdks/python/setup.py Co-authored-by: Danny McCormick <dannymccormick@google.com> --------- Co-authored-by: Danny McCormick <dannymccormick@google.com>
* Changes SplittableDoFn to call TruncateRestriction first when getting a timer caused by drain. We then pass the residual restriction (if present) to ProcessElement. * Adds the rest of a comment string.
…ale graph in notebooks (#39161) * Fix Interactive Beam caching deadlock, race conditions, and stale graph in Colab. - Resolve self-deadlock in background thread: Updated `_wait_for_dependencies` to exclude target PCollections from the wait list when called from the background thread (i.e. when `async_result` is provided), allowing it to only wait on upstream dependencies. - Fix duplicate execution race condition: Replaced waiting on `future.result()` with a `threading.Event` (`_completed_event`) set at the very end of `_on_done`. This ensures `collect()` only resumes after the background job has fully marked the PCollection as computed. - Recalculate uncomputed PCollections: Added a check in `record()` to re-evaluate computed PCollections after waiting, preventing the launch of duplicate pipeline fragments for PCollections that completed during the wait. - Fix stale pipeline graph: Removed caching of the `PipelineGraph` in `RecordingManager`. Re-creating the graph dynamically ensures that new transforms added in subsequent Colab cells are correctly detected and computed. * add unit tests * fix broken unit tests * Apply thread lock following gemini-code-assist suggestions * Address reviewer's comments * Address reviewer's comment * Addressed reviewer's comments * fix broken unit tests and checks * fix formatting issue
…39064) * [IcebergIO] Bump Java floor to 17 (prep for Iceberg 1.11.0) Iceberg 1.11.0 is published as Java 17 bytecode (class file version 61); Gradle module metadata declares "org.gradle.jvm.version": 17. The current Java 11 floor on sdks/java/io/iceberg causes Iceberg 1.11.0 artifacts to fail resolution. Raise the floor independently so the version bump is a clean diff. Iceberg 1.10.0 still resolves under Java 17 (10.0 only requires Java 11+). Tracks #38925. * [IcebergIO] Run IcebergIO CI workflows on Java 17 Iceberg 1.11.0 dropped Java 11 support and this branch raised the iceberg module floor to Java 17 (requireJavaVersion VERSION_17). The IO_Iceberg_* workflows call setup-environment-action without a java-version, which defaults to Java 11, so the module would fail to build or be disabled on CI. Pin these workflows to Java 17, matching the Delta and Debezium IO workflows. For #38925 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [IcebergIO] Raise Java 17 floor for IcebergIO's Java 11 dependents Bumping the iceberg module to Java 17 (Iceberg 1.11.0 dropped Java 11) breaks the Java 11 modules that depend on it, because Gradle's JVM-version variant resolution refuses to let a Java 11 consumer depend on a Java 17 library. Raise requireJavaVersion 11 -> 17 in: - sdks/java/extensions/sql/iceberg - examples/java/iceberg The Java IO expansion service (sdks/java/io/expansion-service) also depends on IcebergIO but was already raised to Java 17 on master in For #38925 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [IcebergIO] Fix Checker skipUses regex dropped on forked Java 17 compile Raising the Java 17 floor routes IcebergIO through a forked compile: on an older host JDK (e.g. Java 11 CI) the module is compiled by a separate Java 17 javac launched via java17Home, and Gradle passes its arguments through an @argfile. The backslash-escaped '\.' in Beam's Checker Framework -AskipUses and -AskipDefs regexes does not survive that @argfile round-trip, so '^org\.slf4j\.Logger.*' becomes a literal-backslash regex that matches nothing and the org.slf4j.Logger nullness suppression is silently dropped. Two latent Logger.info(@nullable) calls in IcebergIO then hard-error under the NullnessChecker. Use the backslash-free character class '[.]' (semantically identical to '\.') so the regexes survive the @argfile round-trip. This fixes the suppression for every module that forks to a newer JDK, with no behavior change on the in-process compile path. * [IcebergIO] Run xlang wrapper validation expansion service on Java 17 The cross-language wrapper validation launches the io expansion-service jar from Python via JAVA_HOME. That jar bundles IcebergIO, now Java 17 bytecode, so launching it on the default Java 11 fails with UnsupportedClassVersionError. createCrossLanguageUsingJavaExpansionTask was missing the JAVA_HOME redirect that #38974 added to the other cross-language task factories. Add it, driven by -PtestJavaVersion (resolving java\Home), and pass -PtestJavaVersion=17 in the Xlang_Generated_Transforms precommit so the expansion service runs on a JDK that can load the bundled Java 17 IOs. * Trigger Python Xlang IO postcommits * [IcebergIO] Run Python Xlang IO postcommits on Java 17 * [IcebergIO] Pin Java 17 on IO expansion service test suites * [IcebergIO] Address Java 17 review comments * [IcebergIO] Raise Iceberg integration tests to Java 17 --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
java
python
io
runners
core
build
examples
extensions
sql
labels
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )