Repository navigation
feat(dataproxy): add queue to UploadInputsRequest - #8148
Merged
Merged
Conversation
…equests Lets a client name the queue the run will be placed on so the server can write run artifacts into that queue's cluster-pool object store instead of the project/domain default pool. Mirrors SelectClusterRequest.queue_id (#8088). Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai>
The previous commit was generated with buf 1.73; CI runs 1.58, which orders source-code-info differently in the embedded descriptor set. Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai>
…queue # Conflicts: # flyteidl2/dataproxy/dataproxy_service.proto # gen/go/flyteidl2/dataproxy/dataproxy_service.pb.go # gen/python/flyteidl2/dataproxy/dataproxy_service_pb2.py # gen/rust/src/flyteidl2.dataproxy.rs # gen/ts/flyteidl2/dataproxy/dataproxy_service_pb.ts
SVilgelm
changed the title
kumare3
added a commit
to flyteorg/flyte-sdk
that referenced
this pull request
Oct 27, 2026
## Summary Builds on #1700 (run-time routing of the code bundle and inputs via `flyte._upload_queue`). This PR now carries only what #1700 does not cover, all on top of the same `upload_queue(...)` context: - **`UploadInputsRequest.queue`**: when an upload queue is set, `upload_inputs` and `upload_trigger` name it on the request, so a control plane that proxies the upload writes the inputs to that queue's pool. Feature-detected on the descriptor; takes effect once a flyteidl2 release includes flyteorg/flyte#8148. - **Trigger uploads routed by queue**: `upload_trigger` now uses the queue-aware resolver, and `offload_trigger_inputs(queue=...)` uploads inside `upload_queue(queue)`. - **`flyte deploy --queue`** / `flyte.deploy(queue=...)`: each environment's effective queue is the override, else its own queue, else the task's. The code bundle is uploaded once per distinct queue, each task is serialized against the bundle for its queue, and each trigger's offloaded inputs go to (and its RunSpec records) the trigger's queue. - **Rerun / run-from-trigger**: when the run's queue differs from the source run's or trigger's queue, the stored inputs are read back and uploaded again via the new queue instead of reusing a URI in another pool's bucket. Same queue keeps reusing it. Ref RUN-70. Related: flyteorg/flyte#8148 (IDL), flyteorg/flyte#8088 / unionai cloud SelectCluster `queue_id`. ## Compatibility with #1714 #1714 routes the image-build run to the run's queue by adding `queue=` to `build_images` / `_build_images` / `_build_image_bg`. This PR does not touch those functions or their callers in `_run.py`, so the two compose. For `flyte deploy`, #1714's per-build `queue=` can take each environment's effective queue computed here. ## Test plan - `pytest tests/flyte/remote/test_cluster_aware_dataproxy.py tests/flyte/remote/test_trigger_create_offload.py tests/flyte/remote/test_data_errors.py tests/flyte/test_run_runspec_chars.py tests/flyte/code_bundle tests/flyte/test_deploy.py tests/flyte/test_rerun.py tests/flyte/test_run_trigger.py tests/flyte/cli/test_deploy_ctx_obj.py`: 273 passed, 1 skipped. - `ruff check`, `ruff format --check`, `mypy` clean on touched files. Signed-off-by: Sergey Vilgelm <sergey@union.ai> Co-authored-by: Ketan Umare <kumare3@users.noreply.github.com>
kumare3
approved these changes
Oct 27, 2026
kumare3
added a commit
to flyteorg/flyte-sdk
that referenced
this pull request
Oct 27, 2026
## Summary - Bumps `flyteidl2` 2.0.52 → 2.0.53, which carries `UploadInputsRequest.queue` (flyteorg/flyte#8148). - #1671 set that field only when the installed flyteidl2 had it, so on 2.0.52 upload-queue routing never reached `UploadInputs`/`UploadTrigger` requests. With the pin guaranteeing the field, `_set_upload_inputs_queue` drops the descriptor check and the fake-descriptor test. Server side still pending: unionai/cloud#19052 (dataproxy honors the queue). ## Test plan - `pytest tests/flyte/remote tests/flyte/test_deploy.py tests/flyte/test_rerun.py tests/flyte/test_run_trigger.py tests/flyte/cli/test_deploy_ctx_obj.py` — 876 passed 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Ketan Umare <kumare3@users.noreply.github.com> Co-authored-by: Ketan Umare <kumare3@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
EngHabu
added a commit
that referenced
this pull request
Oct 29, 2026
The branch was 63 commits behind and GitHub had marked the PR DIRTY, which is why only DCO/WIP/label/snyk ran on the last push: a `pull_request` workflow needs a computable merge commit, so `synchronize` skipped Go Tests and Check Generated Files entirely. Green on four of twenty checks is not green. All four conflicts were in GENERATED files -- the Go, Python, Rust and TS dataproxy stubs -- because main's #8148 and #8216 added `queue` to the upload requests while this branch reshaped VolumeData. The `.proto` itself merged cleanly, so the resolution is not a hand-merge of generated code: regenerate. Ran buf 1.47.2 over all four templates and staged the output, then checked that both changes survived -- ActionOutput/ArtifactVersion present in all four languages, and main's `queue` present in the Go stub. `gen/python/uv.lock`, `gen/ts/package-lock.json` and the mockery-generated `*/mocks/*` are restored rather than committed: `--clean` removes them and CI recreates them from `uv sync --all-groups --frozen`, `npm install` and `make check-mocks`, and this merge alters no dependency. Checked: buf lint, buf format -d on the changed proto, no conflict markers anywhere under gen/, and `go build ./gen/go/flyteidl2/dataproxy/...`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EuLd1EfGEZ4xf9FEq6BHNr Signed-off-by: Haytham Abuelfutuh <haytham@union.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking issue
Related to #8088 (SelectCluster by
queue_id) and #8216 (CreateUploadLocationRequest.queue). Linear: RUN-70.Why are the changes needed?
A client that already knows which queue a run will be placed on has no way to say so when it uploads the run's inputs. The server therefore writes those artifacts into the project/domain default cluster pool's bucket, which is the wrong bucket when the queue maps to a different pool. Server-side routing by queue already exists for
SelectCluster(#8088); #8216 added the hint toCreateUploadLocation; this adds it toUploadInputsfor backends that are not routed throughSelectCluster.What changes were proposed in this pull request?
In
flyteidl2/dataproxy/dataproxy_service.proto:UploadInputsRequest: new optionalstring queue = 8. The name of the queue the run will be placed on, resolved in the run's project/domain scope likeRunSpec.queue. When set, inputs are written to that queue's cluster-pool object store; when empty, the project/domain default queue's pool is used.Single-cluster deployments may ignore the field. Generated Go, Python, Rust and TypeScript outputs are regenerated. No validation rules; purely additive and wire-compatible.
Consumers:
unionai/clouddataproxy (server side) andflyte-sdk(client side).How was this patch tested?
make buf-lint,buf-go,buf-python,buf-rust,buf-tsrun locally; only dataproxy outputs changed.go build ./...andgo vet ./gen/go/flyteidl2/dataproxy/...pass.ghcr.io/flyteorg/flyte/ci:v2) socheck-generatematches.Labels
Check all the applicable boxes