diff --git a/docs/content/_navigation.json b/docs/content/_navigation.json index ea1a8211f48c5..d62dba54dbdc0 100644 --- a/docs/content/_navigation.json +++ b/docs/content/_navigation.json @@ -556,6 +556,10 @@ { "title": "Part 1: Setting up an AKS agent.", "path": "/dagster-plus/deployment/azure/aks-agent" + }, + { + "title": "Part 2: Deploy user code using ACR.", + "path": "/dagster-plus/deployment/azure/acr-user-code" } ] } diff --git a/docs/content/dagster-plus/deployment/azure/acr-user-code.mdx b/docs/content/dagster-plus/deployment/azure/acr-user-code.mdx new file mode 100644 index 0000000000000..00af5e80c1d58 --- /dev/null +++ b/docs/content/dagster-plus/deployment/azure/acr-user-code.mdx @@ -0,0 +1,135 @@ +# Deploying user code in Azure Container Registry (ACR) with Dagster+ + +This quickstart guide will walk you through setting up a new repository for your Dagster code, setting up CI/CD with GitHub Actions backed by Azure Container Registry (ACR), and deploying your code to your Azure Kubernetes Service (AKS) cluster. + +This guide assumes you already have an AKS agent running. You can follow along [here](/dagster-plus/deployment/azure/aks-agent) if you still need to set up an AKS agent. + +## Prerequisites + +This guide will use a Github repository to store the Dagster code, and GitHub Actions to deploy the code to Azure Container Registry - but you should be able to adapt these steps to other version control systems and CI/CD tools. + +- The azure CLI installed on your machine. You can download it [here](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli). +- A GitHub account, and the ability to run GitHub Actions workflows in a repository. + +## Step 1: Creating a repository for Dagster code. + +We'll create a new repository based on the [Dagster+ hybrid quickstart repository](https://github.com/dagster-io/dagster-cloud-hybrid-quickstart). We'll go through these steps using a brand new repository in GitHub, but you should be able to easily adapt these steps to an existing repository or other version control systems. + +First, [create a new repository in GitHub](https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-new-repository). Going forward, we'll refer to this repository as `dagster-plus-code`. + +Next, we'll clone both our new repository and the Dagster+ hybrid quickstart repository to our local machine. + +```bash +git clone +git clone git@github.com:dagster-io/dagster-cloud-hybrid-quickstart.git +``` + +We'll copy the contents of the `dagster-cloud-hybrid-quickstart` repository into our `dagster-plus-code` repository, and commit the changes. + +```bash +cp -r dagster-cloud-hybrid-quickstart/* dagster-plus-code/ --exclude=".git" +cd dagster-plus-code +git add . +git commit -m "Initial commit" +git push +``` + +### Project structure + +The project has the following structure: + +```plaintext +├── .github +│ └── workflows +│ └── dagster-cloud-deploy.yml # GitHub Actions workflow for re-deploying code location +├── .vscode # Standard VSCode settings for working with a Dagster repository +├── Dockerfile # Dockerfile for building the user code image +├── README.md +├── dagster_cloud.yaml # Configuration file describing all code locations in the repository +├── pyproject.toml # Python project configuration file for the code location +├── quickstart_etl # Python package containing the user code +│ ├── __init__.py +│ ├── assets +│ │ ├── __init__.py +│ │ └── hackernews.py +│ └── definitions.py +├── quickstart_etl_tests # User code tests +│ ├── __init__.py +│ └── test_assets.py +├── setup.cfg +└── setup.py +``` + +## Step 2: Setting up an Azure Container Registry + +Next, we'll set up an Azure Container Registry to store our Docker images. We'll use the Azure CLI to create the registry. + +```bash +az login +az acr create --resource-group --name dagstercodelocations --sku Basic +``` + +Then, we'll make images from our ACR available to our AKS cluster. + +```bash +az aks update -n -g --attach-acr dagstercodelocations +``` + +## Step 3: Setting up GitHub Actions + +Now, we'll set up a Github Actions workflow to build and push our Docker image to Azure Container Registry. + +We already have a GitHub Actions workflow in our repository, located at `.github/workflows/dagster-cloud-deploy.yml`. This workflow will build the Docker image, push it to ACR, and update the code location in Dagster+. To get it working with your repository, you'll need to do a few things. + +#### Generate Azure credentials + +First, we'll need to generate a service principal for GitHub Actions to use to authenticate with Azure. We'll use the Azure CLI to create the service principal. + +```bash +az ad sp create-for-rbac --name "github-actions-acr" --role contributor --scopes /subscriptions//resourceGroups//providers/Microsoft.ContainerRegistry/registries/ +``` + +This command will output a JSON object with the service principal details. Make sure to save the `appId`, `password`, and `tenant` values - we'll use them in the next step. + +### Add secrets to your repository + +We'll add the service principal details as secrets in our repository. Go to your repository in GitHub, and navigate to `Settings` -> `Secrets`. Add the following secrets: + +- `AZURE_CLIENT_ID`: The `appId` from the service principal JSON object. +- `AZURE_CLIENT_SECRET`: The `password` from the service principal JSON object. + +### Update the workflow + +Finally, we'll update the workflow to use the service principal details. Open `.github/workflows/dagster-cloud-deploy.yml` in your repository, and uncomment the section on Azure Container Registry. It should look like this: + +```yaml +# Azure Container Registry (ACR) +# https://github.com/docker/login-action#azure-container-registry-acr +- name: Login to Azure Container Registry + if: steps.prerun.outputs.result != 'skip' + uses: docker/login-action@v3 + with: + registry: ${{ env.IMAGE_REGISTRY }} + username: ${{ secrets.AZURE_CLIENT_ID }} + password: ${{ secrets.AZURE_CLIENT_SECRET }} +``` + +### Push and run the workflow + +Now, commit and push the changes to your repository. The GitHub Actions workflow should run automatically. You can check the status of the workflow in the `Actions` tab of your repository. + + + +When the workflow completes, you should see the new code location in Dagster+. Navigate to the `Status` page, and click the `Code Locations` tab. You should see your new code location listed. + + diff --git a/docs/content/dagster-plus/deployment/azure/aks-agent.mdx b/docs/content/dagster-plus/deployment/azure/aks-agent.mdx index b86833e959e8e..cb9bf9110ee36 100644 --- a/docs/content/dagster-plus/deployment/azure/aks-agent.mdx +++ b/docs/content/dagster-plus/deployment/azure/aks-agent.mdx @@ -38,3 +38,7 @@ kubectl config current-context ## Step 3: Install the Dagster+ agent on the AKS cluster. Next, we'll install the agent helm chart. You should be able to follow the guide [here](/concepts/deployment/agents/kubernetes/configuring-running-kubernetes-agent) to install the agent on the AKS cluster. + +## Next steps + +Now that you have an agent running on your AKS cluster, you can start deploying Dagster code to it. You can follow the guide [here](/concepts/deployment/azure/acr-user-code) to deploy user code to your AKS cluster backed by Azure Container Registry (ACR). diff --git a/docs/content/dagster-plus/deployment/azure/overview.mdx b/docs/content/dagster-plus/deployment/azure/overview.mdx index 7db144bdf31dc..c9b8cdbc23013 100644 --- a/docs/content/dagster-plus/deployment/azure/overview.mdx +++ b/docs/content/dagster-plus/deployment/azure/overview.mdx @@ -3,8 +3,12 @@ This guide will walk you through deploying Dagster+ on Azure. We'll start from a brand new organization in Dagster+, and finish with a full hybrid deployment of Dagster+ using Azure infrastructure. - + + diff --git a/docs/next/public/images/dagster-cloud/azure/dagster-cloud-code-locations.png b/docs/next/public/images/dagster-cloud/azure/dagster-cloud-code-locations.png new file mode 100644 index 0000000000000..e1b0835bf437b Binary files /dev/null and b/docs/next/public/images/dagster-cloud/azure/dagster-cloud-code-locations.png differ diff --git a/docs/next/public/images/dagster-cloud/azure/github-actions-workflow.png b/docs/next/public/images/dagster-cloud/azure/github-actions-workflow.png new file mode 100644 index 0000000000000..1b6a0fd35f156 Binary files /dev/null and b/docs/next/public/images/dagster-cloud/azure/github-actions-workflow.png differ