Skip to content

Add Modal orchestrator with step operator and orchestrator flavors #3733

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 47 commits into
base: develop
Choose a base branch
from
+2,412 −110
Open

Add Modal orchestrator with step operator and orchestrator flavors #3733

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
a599f69
Add Modal orchestrator with step operator and orchestrator flavors
htahir1 Jun 12, 2025
b122c63
Add Modal Orchestrator Integration
htahir1 Jun 12, 2025
cd6b59f
Add pipeline-wide resource settings for hardware resources
htahir1 Jun 12, 2025
58d72c5
Add Modal step operator orchestrator to run pipelines on Modal
htahir1 Jun 12, 2025
9a0f2a5
Add log streaming for async execution in Modal Orchestrator
htahir1 Jun 12, 2025
7cf8fd5
Add app warming window hours for container reuse
htahir1 Jun 12, 2025
60d227a
Remove unnecessary exception handling in ModalOrchestrator and ModalS…
htahir1 Jun 12, 2025
45ed008
Refactor return statement to assign and return validator
htahir1 Jun 12, 2025
0b6b9f9
Add Modal integration utils module for mypy discoverability
htahir1 Jun 12, 2025
d8318b1
Update Modal integration to require Modal version 1
htahir1 Jun 12, 2025
b7400b6
Introduce ModalExecutionMode enum for execution modes
htahir1 Jun 12, 2025
ee15117
Refactor Modal authentication setup and deployment
htahir1 Jun 12, 2025
85aeef4
Update platform references in log messages
htahir1 Jun 12, 2025
01f7ef2
Include pipeline run ID for isolation and conflict prevention
htahir1 Jun 13, 2025
6deba69
Merge remote-tracking branch 'origin/develop' into feature/modal-orch…
htahir1 Jun 13, 2025
481972b
Refactor Modal log streaming and improve resource selection
htahir1 Jun 13, 2025
12ac7c4
Clean up Modal orchestrator imports and code organization
htahir1 Jun 13, 2025
788ae84
Improve Modal integration import consistency and requirements
htahir1 Jun 13, 2025
8cdb3c2
Comprehensive Modal orchestrator documentation improvements
htahir1 Jun 13, 2025
f2218b1
Refactor error message formatting and GPU count handling
htahir1 Jun 13, 2025
e9e2574
Add base image requirements and GPU configuration details
htahir1 Jun 13, 2025
74bf8ca
Merge branch 'develop' into feature/modal-orchestrator
htahir1 Jun 16, 2025
331a164
Update src/zenml/integrations/modal/utils.py
safoinme Jun 17, 2025
2770359
Merge remote-tracking branch 'origin/develop' into feature/modal-orch…
htahir1 Jun 18, 2025
f3c0159
Merge remote-tracking branch 'origin/develop' into feature/modal-orch…
htahir1 Jun 23, 2025
3173f6a
Apply suggestions from code review
htahir1 Jun 23, 2025
37d4a57
Update environment variable names to be consistent.- Rename `environm…
htahir1 Jun 23, 2025
e8b6b42
Refactor ModalOrchestrator app naming for image builds
htahir1 Jun 23, 2025
32a56df
Use orchestrator run ID for complete isolation during testing
htahir1 Jun 23, 2025
7616eef
Refactor modal orchestrator for new app-function architecture
htahir1 Jun 23, 2025
3e7fbc5
Refactor Modal orchestrator for better efficiency
htahir1 Jun 23, 2025
5f22bb0
Update environment variable naming convention in Modal step operator
htahir1 Jun 24, 2025
376d7e4
Update Modal orchestrator to use sandbox architecture
htahir1 Jun 24, 2025
04d0204
Update Modal orchestrator for per-step sandboxes
htahir1 Jun 24, 2025
34f717a
Add generate_sandbox_tags function and set tags in sandboxes
htahir1 Jun 24, 2025
bc1f488
Refactor Modal orchestrator and image building functions
htahir1 Jun 24, 2025
32b5571
Added Modal step operator setup guide and examples
htahir1 Jun 24, 2025
31b2c13
Refactor configuration options inheritance logic
htahir1 Jun 24, 2025
4a90c3d
Update modal environment flag format in step operators
htahir1 Jun 24, 2025
312d490
Update modal orchestrator and step operator flavors
htahir1 Jun 24, 2025
1392ec0
Merge branch 'develop' into feature/modal-orchestrator
htahir1 Jun 24, 2025
f9c0818
Merge remote-tracking branch 'origin/develop' into feature/modal-orch…
htahir1 Jun 24, 2025
11d63b3
Update entrypoint configuration description
htahir1 Jun 24, 2025
75e0615
Update configuration section heading to "Configuration Examples".
htahir1 Jun 25, 2025
4fa278f
Create deep copy of settings for Modal orchestrator
htahir1 Jun 25, 2025
79a449a
Add ModalOrchestrator for running pipelines on Modal platform
htahir1 Jun 26, 2025
a499755
Merge remote-tracking branch 'origin/develop' into feature/modal-orch…
htahir1 Jun 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/book/component-guide/orchestrators/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ Out of the box, ZenML comes with a `local` orchestrator already part of the defa
| [SkypilotGCPOrchestrator](skypilot-vm.md) | `vm_gcp` | `skypilot[gcp]` | Runs your pipelines in GCP VMs using SkyPilot |
| [SkypilotAzureOrchestrator](skypilot-vm.md) | `vm_azure` | `skypilot[azure]` | Runs your pipelines in Azure VMs using SkyPilot |
| [HyperAIOrchestrator](hyperai.md) | `hyperai` | `hyperai` | Runs your pipeline in HyperAI.ai instances. |
| [ModalOrchestrator](modal.md) | `modal` | `modal` | Runs your pipelines on Modal's serverless cloud platform. |
| [Custom Implementation](custom.md) | _custom_ | | Extend the orchestrator abstraction and provide your own implementation |

If you would like to see the available flavors of orchestrators, you can use the command:
Expand Down
Loading
Loading