[pull] master from apache:master - #501
Merged
Merged
Conversation
Previously, in ExecutorServiceParallelExecutor, if an exception occurred during registry cleanup (such as a timeout inside DoFn teardown), the pipeline state was transitioned to terminal before the exception was queued in `visibleUpdates`. This introduced a race condition where `waitUntilFinish()` could detect the terminal state and exit successfully before the exception was offered to the updates queue, swallowing the exception. This caused tests like `CallTest.givenTeardownTimeout_throwsError` to fail since they expected the pipeline to throw an exception. This change swaps the order so that the exception is posted to `visibleUpdates` before updating the pipeline state to terminal, ensuring the exception is always propagated.
Update beam-master container
Fix race condition in DirectRunner executor shutdown
…ms. (#39098) * Fix JsonToRow swallowing downstream errors when runners fuse transforms. Separate JSON parsing from MultiOutputReceiver output in ParseWithError so exceptions from fused downstream consumers are not misreported as parse failures. Fixes #20935. * Address review: use MapElements in JsonToRow regression test. Avoid anonymous DoFn capturing the non-serializable test instance so the test is safe on runners that enforce DoFn serialization. * Fix JsonToRow regression test for runner integration suites. Use a static DoFn to avoid serialization issues, set row schema on the downstream transform, and tag the test with ValidatesRunner so it can run via Dataflow validatesRunner tasks. Co-authored-by: Cursor <cursoragent@cursor.com> * Merge apache/beam master and fix JsonToRowTest for CI. Rebase onto current master (1141 commits behind) and remove ValidatesRunner category so the regression test only runs on DirectRunner needsRunnerTests. Suppress UnusedVariable on ThrowingDownstreamDoFn for Error Prone. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )