Skip to content

Commit

Permalink
Move misc docs needed for redirects (#27226)
Browse files Browse the repository at this point in the history 
## Summary & Motivation

Move misc docs needed for redirects.

## How I Tested These Changes

## Changelog

> Insert changelog entry or delete this section.

---------

Signed-off-by: nikki everett <[email protected]>
  • Loading branch information
@neverett
neverett authored Jan 21, 2025
1 parent be38eac commit 7911ccb
Show file tree
Hide file tree
Showing 77 changed files with 2,044 additions and 39 deletions.
384 changes: 384 additions & 0 deletions ...docs/dagster-plus/deployment/management/managing-multiple-projects-and-teams.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,384 @@
---
title: "Managing multiple projects and teams with Dagster+ Hybrid"
description: "How to set up and manage multiple projects with Dagster+ Hybrid deployments."
---


In this guide, we'll cover some strategies for managing multiple projects/code bases and teams in a Dagster+ account.

## Separating code bases

:::note

In this section, repository refers to a version control system, such as Git or Mercurial.

:::

If you want to manage complexity or divide your work into areas of responsibility, consider isolating your code bases into multiple projects with:

- Multiple directories in a single repository, or
- Multiple repositories

Refer to the following table for more information, including the pros and cons of each approach.

<table
className="table"
style={{
width: "100%",
}}
>
<thead>
<tr>
<th
style={{
width: "14%",
}}
>
Approach
</th>
<th
style={{
width: "43%",
}}
>
Multiple directories in a single repository
</th>
<th>Multiple repositories</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<strong>How it works</strong>
</td>
<td>
You can use a single repository to manage multiple projects by placing
each project in a separate directory. Depending on your VCS, you may be
able to set{" "}
<a href="https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners">

Check warning on line 58 in docs/docs-beta/docs/dagster-plus/deployment/management/managing-multiple-projects-and-teams.md

View workflow job for this annotation

GitHub Actions / deploy 

Do not use an `<a>` element to navigate. Use the `<Link />` component from `@docusaurus/Link` instead. See: https://docusaurus.io/docs/docusaurus-core#link
code owners
</a>{" "}
to restrict who can modify each project.
</td>
<td>
For stronger isolation, you can use multiple repositories to manage
multiple projects.
</td>
</tr>
<tr>
<td>
<strong>Pros</strong>
</td>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Simple to implement
</li>
<li>Facilitates code sharing between projects</li>
</ul>
</td>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Stronger isolation between projects and teams
</li>
<li>
Each project has its own CI/CD pipeline and be deployed
independently
</li>
<li>Dependencies between projects can be managed independently</li>
</ul>
</td>
</tr>
<tr>
<td>
<strong>Cons</strong>
</td>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
All projects share the same CI/CD pipeline and cannot be deployed
independently
</li>
<li>
Shared dependencies between projects may cause conflicts and require
coordination between teams
</li>
</ul>
</td>
<td>
Code sharing between projects require additional coordination to publish
and reuse packages between projects
</td>
</tr>
</tbody>
</table>

### Deployment configuration

Whether you use a single repository or multiple, you can use a [`dagster_cloud.yaml` file](/dagster-plus/deployment/code-locations/dagster-cloud-yaml) to define the code locations to deploy. For each repository, follow the [steps appropriate to your CI/CD provider](/dagster-plus/features/ci-cd/configuring-ci-cd) and include only the code locations that are relevant to the repository in your CI/CD workflow.

#### Example with GitHub CI/CD on Hybrid deployment

1. **For each repository**, use the CI/CD workflow provided in [Dagster+ Hybrid quickstart repository](https://github.com/dagster-io/dagster-cloud-hybrid-quickstart/blob/main/.github/workflows/dagster-cloud-deploy.yml).

2. **For each project in the repository**, configure a code location in the [`dagster_cloud.yaml` file](/dagster-plus/deployment/code-locations/dagster-cloud-yaml):

```yaml
# dagster_cloud.yml

locations:
- location_name: project_a
code_source:
package_name: project_a
build:
# ...
- location_name: project_b
code_source:
package_name: project_b
build:
# ...
```

3. In the repository's `dagster-cloud-deploy.yml` file, modify the CI/CD workflow to deploy all code locations for the repository:

```yaml
# .github/workflows/dagster-cloud-deploy.yml

jobs:
dagster-cloud-deploy:
# ...
steps:
- name: Update build session with image tag for "project_a" code location
id: ci-set-build-output-project-a
if: steps.prerun.outputs.result != 'skip'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci set-build-output --location-name=project_a --image-tag=$IMAGE_TAG"

- name: Update build session with image tag for "project_b" code location
id: ci-set-build-output-project-b
if: steps.prerun.outputs.result != 'skip'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci set-build-output --location-name=project_b --image-tag=$IMAGE_TAG"
# ...
```

## Isolating execution context between projects

Separating execution context between projects can have several motivations:

- Facilitating separation of duty between teams to prevent access to sensitive data
- Differing compute environments and requirements, such as different architecture, cloud provider, etc.
- Reducing impact on other projects. For example, a project with a large number of runs can impact the performance of other projects.

In order from least to most isolated, there are three levels of isolation:

- [Code location](#code-location-isolation)
- [Agent](#agent-isolation)
- [Deployment](#deployment-isolation)

### Code location isolation

If you have no specific requirements for isolation beyond the ability to deploy and run multiple projects, you can use a single agent and deployment to manage all your projects as individual code locations.

![Diagram of isolation at the code location level](/images/dagster-cloud/managing-deployments/isolation-level-code-locations.png)

<table
className="table"
style={{
width: "100%",
}}
>
<thead>
<tr>
<th
style={{
width: "50%",
}}
>
Pros
</th>
<th>Cons</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Simplest and most cost-effective solution
</li>
<li>User access control can be set at the code location level</li>
<li>Single glass pane to view all assets</li>
</ul>
</td>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
No isolation between execution environments
</li>
</ul>
</td>
</tr>
</tbody>
</table>

### Agent isolation

:::note

Agent queues are a Dagster+ Pro feature available on hybrid deployment.

:::

Using the [agent routing feature](/dagster-plus/deployment/deployment-types/hybrid/multiple#routing-requests-to-specific-agents), you can effectively isolate execution environments between projects by using a separate agent for each project.

Motivations for utilizing this approach could include:

- Different compute requirements, such as different cloud providers or architectures
- Optimizing for locality or access, such as running the data processing closer or in environment with access to the storage locations

![Diagram of isolation at the agent level](/images/dagster-cloud/managing-deployments/isolation-level-agents.png)

<table
className="table"
style={{
width: "100%",
}}
>
<thead>
<tr>
<th
style={{
width: "50%",
}}
>
Pros
</th>
<th>Cons</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Isolation between execution environments
</li>
<li>User access control can be set at the code location level</li>
<li>Single glass pane to view all assets</li>
</ul>
</td>
<td>Extra work to set up additional agents and agent queues</td>
</tr>
</tbody>
</table>

### Deployment isolation

:::note

Multiple deployments are only available in Dagster+ Pro.

:::

Of the approaches outlined in this guide, multiple deployments are the most isolated solution. The typical motivation for this isolation level is to separate production and non-production environments. It may be considered to satisfy other organization specific requirements.

![Diagram of isolation at the Dagster+ deployment level](/images/dagster-cloud/managing-deployments/isolation-level-deployments.png)

<table
className="table"
style={{
width: "100%",
}}
>
<thead>
<tr>
<th
style={{
width: "50%",
}}
>
Pros
</th>
<th>Cons</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Isolation between assets and execution environments
</li>
<li>
User access control can be set at the code location and deployment
level
</li>
</ul>
</td>
<td>
No single glass pane to view all assets (requires switching between
multiple deployments in the UI)
</td>
</tr>
</tbody>
</table>
Loading

1 comment on commit 7911ccb

@github-actions
Copy link

@github-actions github-actions bot commented on 7911ccb Jan 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs-beta ready!

✅ Preview
https://dagster-docs-beta-6pnpmr63v-elementl.vercel.app

Built with commit 7911ccb.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.