WIP: Azure application gateway Ingress controller

azure/README.md

@@ -16,6 +16,8 @@
 * Public IP
 * Route Table
 * Route Table association
+* Application gateway
+* Role assignments
 
 ## Project directory Structure
 <pre>
@@ -169,3 +171,121 @@ or
 ```console
 terraform destroy -auto-approve
 ```
+
+## Setup Ingress controller
+After `terraform apply` an application gateway will be configured
+
+Get the Kubernetes configuration and access credentials from the Azure using the Azure CLI command.
+
+```console
+az aks get-credentials --name <REPLACEME_AksCluserName>  --resource-group <REPLACEME_ResourceGroupName>
+```
+eg:- az aks get-credentials --name dev-k8ssandra-eks-cluster  --resource-group dev-k8ssandra-resource-group
+
+verify the health and status of the cluster nodes
+
+```console
+kubectl get nodes
+```
+### Install Azure AD Pod Identity
+Azure Active Directory Pod Identity provides token-based access to Azure Resource Manager.
+
+AAD Pod Identity enables Kubernetes applications to access cloud resources securely with Azure Active Directory.
+
+Using Kubernetes primitives, administrators configure identities and bindings to match pods. Then without any code modifications, your containerized applications can leverage any resource in the cloud that depends on AAD as an identity provider.
+
+If RBAC is enabled, run the following command to install Azure AD Pod Identity to your cluster:
+
+```console
+kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml
+```
+If RBAC is disabled, run the following command to install Azure AD Pod Identity to your cluster:
+
+```console
+kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment.yaml
+```
+
+### Install Helm
+The code in this section uses Helm - Kubernetes package manager - to install the application-gateway-kubernetes-ingress package:
+
+Run the follow helm commands to add the AGIC Helm repository:
+
+```console
+helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
+helm repo update
+```
+
+### Install Ingress Controller Helm Chart
+Locate the [helm-config.yaml](./scripts/helm-config.yaml) file in the scripts folder.
+
+The values are described as follows:
+
+**verbosityLevel:** Sets the verbosity level of the AGIC logging infrastructure. See Logging Levels for possible values.
+**appgw.subscriptionId:** The Azure Subscription ID for the App Gateway. Example: a123b234-a3b4-557d-b2df-a0bc12de1234
+**appgw.resourceGroup:** Name of the Azure Resource Group in which App Gateway was created.
+**appgw.name:** Name of the Application Gateway. Example: applicationgateway1.
+**appgw.shared:** This boolean flag should be defaulted to false. Set to true should you need a Shared App Gateway.
+**kubernetes.watchNamespace:** Specify the name space, which AGIC should watch. The namespace can be a single string value, or a comma-separated list of namespaces. Leaving this variable commented out, or setting it to blank or empty string results in Ingress Controller observing all accessible namespaces.
+**armAuth.type:** A value of either aadPodIdentity or servicePrincipal.
+**armAuth.identityResourceID:** Resource ID of the managed identity.
+**armAuth.identityClientId:** The Client ID of the Identity.
+**armAuth.secretJSON:** Only needed when Service Principal Secret type is chosen (when armAuth.type has been set to servicePrincipal).
+
+**Key notes:**
+
+The <identityResourceID> value is created in the terraform script and can be found by running: 
+```console
+echo "$(terraform output identity_resource_id)"
+```
+The <identityClientID> value is created in the terraform script and can be found by running: 
+```console
+echo "$(terraform output identity_client_id)"
+```
+The <resource-group> value is the resource group of your App Gateway or it can be found by running:
+```console
+echo "$(terraform output resource_group)"
+```
+
+The <identity-name> value is the name of the created identity.
+```console
+echo "$(terraform output application_gateway_id)"
+```
+
+All identities for a given subscription can be listed using: az identity list. or it can be found by running: 
+```console
+echo "$(terraform output subcription_id)"
+```
+
+**Install the Application Gateway ingress controller package:**
+Update the required variables in the `helm-config.yaml` file then run the following command, you can find all required values to update this file in the terraform outputs.
+```console
+helm install -f helm-config.yaml application-gateway-kubernetes-ingress/ingress-azure --generate-name
+```
+
+## Install K8ssandra on the Azure Kubernetes Service
+
+**Check the storage classes type by running the following command:**
+check all the storage classes available in the Azure Kubernetes Cluster
+```console
+kubectl get storageclass
+```
+
+**Run the following command to add the k8ssandra helm repository:**
+```console
+helm repo add k8ssandra https://helm.k8ssandra.io/stable
+helm repo update
+```
+
+**Run the following command to deploy the K8ssandra by using Helm command:**
+```console
+helm install <REPLACEME_release-name> k8ssandra/k8ssandra --set cassandra.cassandraLibDirVolume.storageClass=default
+```
+eg:- helm install test k8ssandra/k8ssandra --set cassandra.cassandraLibDirVolume.storageClass=default
+
+**Create Nodeport & Ingress service:**
+[nodeport.yaml](./scripts/nodeport.yaml) and [ingress.yaml](./scripts/nodeport) file will be found in the scripts folder.
+run the following command to create Nodeport service and Ingress.
+```console
+kubectl create -f ./nodeport.yaml
+kubectl create -f ./ingress.yaml
+```

azure/env/README.md

@@ -119,25 +119,29 @@ module "storage" {
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | <a name="input_environment"></a> [environment](#input\_environment) | Name of the environment where infrastructure being built. | `string` | n/a | yes |
-| <a name="input_kubernetes_version"></a> [kubernetes\_version](#input\_kubernetes\_version) | version of the kubernetes cluster | `string` | `"1.19.9"` | no |
+| <a name="input_kubernetes_version"></a> [kubernetes\_version](#input\_kubernetes\_version) | Version of the Azure kubernetes cluster | `string` | `"1.19.9"` | no |
 | <a name="input_max_count"></a> [max\_count](#input\_max\_count) | Maximum Node Count | `number` | `5` | no |
 | <a name="input_min_count"></a> [min\_count](#input\_min\_count) | Minimum Node Count | `number` | `3` | no |
-| <a name="input_name"></a> [name](#input\_name) | AKS name in Azure | `string` | n/a | yes |
-| <a name="input_node_count"></a> [node\_count](#input\_node\_count) | Number of AKS worker nodes | `number` | `5` | no |
+| <a name="input_name"></a> [name](#input\_name) | Name is the prefix to use for resources that needs to be created. | `string` | n/a | yes |
+| <a name="input_node_count"></a> [node\_count](#input\_node\_count) | Number of AKS worker nodes | `number` | `3` | no |
 | <a name="input_private_service_endpoints"></a> [private\_service\_endpoints](#input\_private\_service\_endpoints) | service endpoints to attach Private Subnets. | `list(string)` | <pre>[<br>  "Microsoft.Storage"<br>]</pre> | no |
-| <a name="input_private_subnet_prefixes"></a> [private\_subnet\_prefixes](#input\_private\_subnet\_prefixes) | value | `list(string)` | <pre>[<br>  "10.1.1.0/24"<br>]</pre> | no |
+| <a name="input_private_subnet_prefixes"></a> [private\_subnet\_prefixes](#input\_private\_subnet\_prefixes) | Private subnet address prefixes. | `list(string)` | <pre>[<br>  "10.1.1.0/24"<br>]</pre> | no |
 | <a name="input_public_service_endpoints"></a> [public\_service\_endpoints](#input\_public\_service\_endpoints) | service endpoints to attche public Subnets. | `list(string)` | `[]` | no |
-| <a name="input_public_subnet_prefixes"></a> [public\_subnet\_prefixes](#input\_public\_subnet\_prefixes) | value | `list(string)` | <pre>[<br>  "10.1.0.0/24"<br>]</pre> | no |
+| <a name="input_public_subnet_prefixes"></a> [public\_subnet\_prefixes](#input\_public\_subnet\_prefixes) | Public subnet address prefixes. | `list(string)` | <pre>[<br>  "10.1.0.0/24"<br>]</pre> | no |
 | <a name="input_region"></a> [region](#input\_region) | Azure location where all the resources being created. | `string` | n/a | yes |
-| <a name="input_system_node_count"></a> [system\_node\_count](#input\_system\_node\_count) | Number of AKS worker nodes | `number` | `3` | no |
-| <a name="input_vm_size"></a> [vm\_size](#input\_vm\_size\_count) | Specifies the size of the virtual machine. | `string` | `Standard_E8_v4` | no |
+| <a name="input_resource_owner"></a> [resource\_owner](#input\_resource\_owner) | The name of the Account Owner | `string` | n/a | yes |
+| <a name="input_vm_size"></a> [vm\_size](#input\_vm\_size) | Specifies the size of the virtual machine. | `string` | `"Standard_E8_v4"` | no |
 
 ## Outputs
 
 | Name | Description |
 |------|-------------|
 | <a name="output_aks_fqdn"></a> [aks\_fqdn](#output\_aks\_fqdn) | Azure kuberenetes service fqdn. |
 | <a name="output_aks_id"></a> [aks\_id](#output\_aks\_id) | Azure kuberenetes service id. |
+| <a name="output_application_gateway_id"></a> [application\_gateway\_id](#output\_application\_gateway\_id) | Azure application gateway ID. |
 | <a name="output_connect_cluster"></a> [connect\_cluster](#output\_connect\_cluster) | Connection string to be used to configure kubectl. |
+| <a name="output_current_subscription_display_name"></a> [current\_subscription\_display\_name](#output\_current\_subscription\_display\_name) | Azure subscription ID. |
+| <a name="output_identity_client_id"></a> [identity\_client\_id](#output\_identity\_client\_id) | User managed identity client ID. |
+| <a name="output_identity_resource_id"></a> [identity\_resource\_id](#output\_identity\_resource\_id) | User managed identity resource ID. |
 | <a name="output_resource_group"></a> [resource\_group](#output\_resource\_group) | The name of the resource group in which the resources will be created. |
 | <a name="output_storage_account_id"></a> [storage\_account\_id](#output\_storage\_account\_id) | Azure Storage account id. |

azure/env/dev.tf

@@ -49,11 +49,13 @@ module "vnet" {
 
 # Azure Identities module
 module "iam" {
-  source      = "../modules/iam"
-  name        = local.prefix
-  environment = var.environment
-  location    = var.region
-  tags        = local.tags
+  source        = "../modules/iam"
+  name          = local.prefix
+  environment   = var.environment
+  public_subnet = module.vnet.public_subnets
+  appgw_id      = module.vnet.appgw_id
+  location      = var.region
+  tags          = local.tags
 }
 
 # Azure Storage Account module

azure/env/outputs.tf

@@ -15,19 +15,19 @@
 # Output attribute of the Azure Kubernetes Cluster id.
 output "aks_id" {
   description = "Azure kuberenetes service id."
-  value = module.aks.azurerm_kubernetes_cluster_id
+  value       = module.aks.azurerm_kubernetes_cluster_id
 }
 
 # Output attribute of the Azure Kubernetes Cluster fqdn.
 output "aks_fqdn" {
   description = "Azure kuberenetes service fqdn."
-  value = module.aks.azurerm_kubernetes_cluster_fqdn
+  value       = module.aks.azurerm_kubernetes_cluster_fqdn
 }
 
 # Output attribute of the Resource Group.
 output "resource_group" {
   description = "The name of the resource group in which the resources will be created."
-  value = module.iam.resource_group_name
+  value       = module.iam.resource_group_name
 }
 
 # Output attribute of the Storage Account id.
@@ -39,5 +39,26 @@ output "storage_account_id" {
 # connection string to connect you Azure Kubernetes cluster.
 output "connect_cluster" {
   description = "Connection string to be used to configure kubectl."
-  value          =  format("az aks get-credentials --resource-group %s --name %s", module.iam.resource_group_name, module.aks.azurerm_kubernetes_cluster_name)
+  value       = format("az aks get-credentials --resource-group %s --name %s", module.iam.resource_group_name, module.aks.azurerm_kubernetes_cluster_name)
+}
+
+# Output variables to update helm-config file for ingress controller set up.
+output "identity_resource_id" {
+  description = "User managed identity resource ID."
+  value       = module.iam.user_id
+}
+
+output "identity_client_id" {
+  description = "User managed identity client ID."
+  value       = module.iam.identity_client_id
+}
+
+output "current_subscription_display_name" {
+  description = "Azure subscription ID."
+  value       = data.azurerm_subscription.current.subscription_id
+}
+
+output "application_gateway_id" {
+  description = "Azure application gateway ID."
+  value       = module.vnet.appgw_id
 }

azure/env/variables.tf

@@ -63,13 +63,13 @@ variable "vm_size" {
 }
 
 variable "public_subnet_prefixes" {
-  description = "value"
+  description = "Public subnet address prefixes."
   type        = list(string)
   default     = ["10.1.0.0/24"]
 }
 
 variable "private_subnet_prefixes" {
-  description = "value"
+  description = "private subnet address prefixes."
   type        = list(string)
   default     = ["10.1.1.0/24"]
 }

azure/modules/aks/main.tf

@@ -58,6 +58,16 @@ resource "azurerm_kubernetes_cluster" "kubernetes_cluster" {
     service_cidr = var.service_cidr
   }
 
+  addon_profile {
+    http_application_routing {
+      enabled = false
+    }
+  }
+
+  role_based_access_control {
+    enabled = var.aks_enable_rbac
+  }
+
   lifecycle {
     # This life cycle policy to prevent the cluster being destroy. It set to false.
     prevent_destroy = false

azure/modules/aks/variables.tf

@@ -95,6 +95,11 @@ variable "service_cidr" {
   type        = string
 }
 
+variable "aks_enable_rbac" {
+  description = "Enable RBAC on the AKS cluster. Defaults to false."
+  default     = "false"
+}
+
 # tags
 variable "tags" {
   description = "A map of the tags to use on the resources that are deployed with this module."

azure/modules/iam/README.md

@@ -37,5 +37,6 @@ No modules.
 |------|-------------|
 | <a name="output_location"></a> [location](#output\_location) | Azure location where all the resources being created. |
 | <a name="output_principal_id"></a> [principal\_id](#output\_principal\_id) | Azure Managed identity principal id. |
+| <a name="output_identity_client_id"></a> [identity\_client\_id](#output\_identity\_client\_id) | Azure Managed identity client id. |
 | <a name="output_resource_group_name"></a> [resource\_group\_name](#output\_resource\_group\_name) | The name of the resource group in which the resources will be created. |
 | <a name="output_user_id"></a> [user\_id](#output\_user\_id) | Azure Managed Identity id. |

azure/modules/iam/main.tf

@@ -21,9 +21,44 @@ resource "azurerm_user_assigned_identity" "user_assigned_identity" {
   tags = var.tags
 }
 
+data "azuread_service_principal" "service_principal" {
+  display_name = "k8ssandra-terraform"
+}
+
 # Azure Resource group
 resource "azurerm_resource_group" "resource_group" {
   name     = format("%s-resource-group", var.name)
   location = var.location
   tags     = var.tags
 }
+
+# Azure network role assignment.
+resource "azurerm_role_assignment" "network_role_assignment" {
+  scope                = var.public_subnet
+  role_definition_name = "Network Contributor"
+  principal_id         = data.azuread_service_principal.service_principal.object_id
+}
+
+# Azure Operator role assignment.
+resource "azurerm_role_assignment" "Operator_role_assignment" {
+  scope                = azurerm_user_assigned_identity.user_assigned_identity.id
+  role_definition_name = "Managed Identity Operator"
+  principal_id         = data.azuread_service_principal.service_principal.object_id
+  depends_on           = [azurerm_user_assigned_identity.user_assigned_identity]
+}
+
+# Azure contributor role assignment.
+resource "azurerm_role_assignment" "contributor_role_assignment" {
+  scope                = var.appgw_id
+  role_definition_name = "Contributor"
+  principal_id         = azurerm_user_assigned_identity.user_assigned_identity.principal_id
+  depends_on           = [azurerm_user_assigned_identity.user_assigned_identity]
+}
+
+# Azure reader role assignment.
+resource "azurerm_role_assignment" "reader_role_assignment" {
+  scope                = azurerm_resource_group.resource_group.id
+  role_definition_name = "Reader"
+  principal_id         = azurerm_user_assigned_identity.user_assigned_identity.principal_id
+  depends_on           = [azurerm_user_assigned_identity.user_assigned_identity]
+}

azure/modules/iam/outputs.tf

@@ -18,6 +18,11 @@ output "user_id" {
   value       = azurerm_user_assigned_identity.user_assigned_identity.id
 }
 
+output "identity_client_id" {
+  description = "Azure Managed identity client id."
+  value = azurerm_user_assigned_identity.user_assigned_identity.client_id
+}
+
 output "principal_id" {
   description = "Azure Managed identity principal id."
   value       = azurerm_user_assigned_identity.user_assigned_identity.principal_id

azure/modules/iam/variables.tf

@@ -28,7 +28,17 @@ variable "environment" {
   type        = string
 }
 
+variable "public_subnet" {
+  description = "The subnet id of the virtual network where the virtual machines will reside."
+  type        = string
+}
+
+variable "appgw_id" {
+  description = "The id of the Application gateway"
+  type        = string
+}
+
 variable "tags" {
   description = "A map of the tags to use on the resources that are deployed with this module."
   type        = map(string)
-}
+}