[azure-docs] compute log manager docs

docs/content/_navigation.json

@@ -560,6 +560,10 @@
                   {
                     "title": "Part 2: Deploy user code using ACR.",
                     "path": "/dagster-plus/deployment/azure/acr-user-code"
+                  },
+                  {
+                    "title": "Part 3: Storing compute logs in Azure Blob Storage / Azure Data Lake",
+                    "path": "/dagster-plus/deployment/azure/blob-compute-logs"
                   }
                 ]
               }

docs/content/dagster-plus/deployment/azure/blob-compute-logs.mdx

@@ -0,0 +1,119 @@
+# Storing compute logs in Azure Blob Storage/Azure Data Lake Storage
+
+In this guide, we'll walk through how to store compute logs in Azure Blob Storage or Azure Data Lake Storage. This guide assumes you have already set up an Azure Kubernetes Service (AKS) agent and deployed user code in Azure Container Registry (ACR).
+
+This guide focuses on using Azure Blob Storage, but the same steps should be applicable for Azure Data Lake Storage.
+
+If you have not yet set up an AKS agent, you can follow the [Deploy an Azure Kubernetes Service (AKS) agent guide](/dagster-plus/deployment/azure/aks-agent).
+If you have not yet deployed user code in ACR, you can follow the [Deploy user code in Azure Container Registry (ACR) guide](/dagster-plus/deployment/azure/acr-user-code).
+
+## Prerequisites
+
+To complete the steps in this guide, you'll need:
+- The Azure CLI installed on your machine. You can download it [here](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli).
+- An Azure account with the ability to create resources in Azure Blob Storage or Azure Data Lake Storage.
+- An Azure container in Azure Blob Storage or Azure Data Lake Storage where you want to store logs.
+
+## Step 1: Give AKS agent access to blob storage account
+
+We need to ensure that the AKS agent has the necessary permissions to write logs to Azure Blob Storage or Azure Data Lake Storage. How
+ We'll do this with some azure CLI commands.
+
+First, we'll enable the cluster to use workload identity. This will allow the AKS agent to use a managed identity to access Azure resources.
+
+```bash
+az aks update --resource-group <resource-group> --name <cluster-name> --enable-workload-identity
+```
+
+Then, we'll create a new managed identity for the AKS agent, and a new service account in our AKS cluster.
+
+```bash
+az identity create --resource-group <resource-group> --name agent-id
+kubectl create serviceaccount dagster-agent-sa --namespace dagster-agent 
+```
+
+Now we need to federate the managed identity with the service account.
+
+```bash
+az identity federated-credential create \                                                                             
+  --name dagster-agent-federated-id \
+  --identity-name agent-id \
+  --resource-group <resource-group> \
+  --issuer $(az aks show -g <resource-group> -n <aks-cluster-name> --query "oidcIssuerProfile.issuerUrl" -otsv) \
+  --subject system:serviceaccount:dagster-agent:dagster-agent-sa
+```
+
+Finally, we'll edit our AKS agent deployment to use the new service account.
+
+```bash
+kubectl edit deployment <your-user-cloud-deployment> -n dagster-agent 
+```
+
+In the deployment manifest, add the following lines:
+
+```yaml
+metadata:
+  ...
+  labels:
+    ...
+    azure.workload.identity/use: "true"
+spec:
+  ...
+  template:
+    ...
+    spec:
+      ...
+      serviceAccountName: dagster-agent-sa
+```
+
+If everything is set up correctly, you should be able to run the following command and see an access token returned:
+```bash
+kubectl exec -n dagster-agent -it <pod-in-cluster> -- bash
+# in the pod
+curl -H "Metadata:true" "http://169.254.169.254/metadata/identity/oauth2/token?resource=https://storage.azure.com/"
+```
+
+## Step 2: Configure Dagster to use Azure Blob Storage
+
+Now, you need to update the helm values to use Azure Blob Storage for logs. You can do this by editing the `values.yaml` file for your user-cloud deployment.
+
+Pull down the current values for your deployment:
+
+```bash
+helm get values user-cloud > current-values.yaml
+```
+
+Then, edit the `current-values.yaml` file to include the following lines:
+
+```yaml
+computeLogs:
+  enabled: true
+  custom:
+    module: dagster_azure.blob.compute_log_manager
+    class: AzureBlobComputeLogManager
+    config:
+      storage_account: chriscomplogmngr
+      container: mycontainer
+      default_azure_credential:
+        exclude_environment_credential: false
+      prefix: dagster-logs-
+      local_dir: "/tmp/cool"
+      upload_interval: 30
+```
+    
+Finally, update your deployment with the new values:
+    
+```bash
+helm upgrade user-cloud dagster-cloud/dagster-cloud-agent -n dagster-agent -f current-values.yaml
+```
+
+## Step 3: Verify logs are being written to Azure Blob Storage
+
+You should be able to kick off a run of `all_assets_job`, and when you go to the stdout/stderr window of the run page, you should see a log file that directs you to the Azure Blob Storage container.
+
+<Image src="/images/dagster-cloud/azure/azure-blob-storage-logs.png" alt="Azure Blob Storage logs in Dagster" width={970} height={794} />
+
+<Note>
+Whether or not the URL will be clickable depends on whether your logs are public or private. If they are private, directly clicking the link would not work, and instead you should use either the Azure CLI or the Azure Portal to access the logs using the URL.
+</Note>
+

docs/content/dagster-plus/deployment/azure/overview.mdx

@@ -11,4 +11,8 @@ This guide will walk you through deploying Dagster+ on Azure. We'll start from a
     title="Step 2: Deploy user code in Azure Container Registry (ACR)"
     href="/dagster-plus/deployment/azure/acr-user-code"
   ></ArticleListItem>
+  <ArticleListItem
+    title="Step 3: Store compute logs in Azure Blob Storage or Azure Data Lake Storage"
+    href="/dagster-plus/deployment/azure/blob-compute-logs"
+  ></ArticleListItem>
 </ArticleList>