Skip to content

Latest commit

 

History

History
235 lines (160 loc) · 13.1 KB

add-environment.md

File metadata and controls

235 lines (160 loc) · 13.1 KB

Add a new environment

Important

If you are adding a new production environment, then you will need to make sure that you are making adjustments to script and github workflow files in a release branch

1. Create github environment

  1. Create github environment:

    1. Add the new environment to github-setup-environments.yml
    2. Go to the Actions tab in the github repo
    3. Manually run the workflow you have adjusted above: Github Create Environments

  2. Add approvals and branch protection policies to github environment

    Go to Settings > Environments in the github repo and select the environment you have just created. Add the branch protection rules as required for the new environment.

    Recommendations:

    • qa: require approval (to allow qa to pull changes into environment at their own pace, and to avoid a deploy from release branch being overwritten)
    • demo, demo-xx: require approval
    • staging, prod, and prod-xx:
      • require approval
      • only allowed to run for release branches and the default (eg master) branch

    For more information see: https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment

2. Authenticate github actions with Azure

  1. Ensure service principal used by github actions can authenticate for the new environment.

    The github actions workflow will authenticate to Azure AD using a service principal using a federated credential for the environment it is attempting to deploy to. Therefore, for any new environment, you need to ensure that the service principal has a federated credential setup for this new environment.

    If the new environment is in a difference subscription from the existing environments, then you will need to:

    1. Create a new service principal. Follow the section "Create automation service principals" as described in the repo service-principal-automate
    2. Modify set-azure-connection-variables.ps1 with the details of the new subscription and service principal client id
    3. Modify cli-permissions.bicep and provision-shared-services.ps1 for the new service principal
    4. Follow the section Grant RBAC management permission for shared services

    For new or exiting service principals, you will need to create a federated credential for the new environment for the service principal that will deploy to the new environment.

    Follow the section "Create github federated credentials" as described in the repo service-principal-automate, making sure to include just the environment you are adding in the list of environments the script will then add a federated credential for.

  2. Adjust azure connection variables

    Review and adjust the list of environments in set-azure-connection-variables.ps1 to include the new environment. This will be used for authentication to Azure AD, and knowing the id of azure subscription for the new environment.

3. Initial setup of deployment scripts and github workflows

  1. Adjust the list of environments supported by infra-as-code deployment scripts

    Add the new environment to get-product-environment-names.ps1

  2. Modify the github workflow that deploys infrastructure

    You will need to add a new github actions job for the new environment.

    If the new environment is:

  3. Modify the github workflow that deploys application code

    You will need to add a new github actions job for the new environment.

    If the new environment is:

    Also, modify __Application Deployment:

    • to include the new environment in the github actions step Map variables.

    Note: not all configuration variables will be known at this stage, in which case add dummy placeholder values that you will then assign in the steps below.

  4. Modify the github workflows that perform other tasks like granting azure access

    Look for the github workflow files that accept an environment as input parameter. Adjust this list to include the new environment. Some of the github workflows that you have adjusted might also need a new job adding for the new environment, some might not.

4. Configure DNS and TLS

Tip

You can defer the DNS and TLS configuration until after the application code has been deployed to the new environment. To do this you will need to adjust the get-product-conventions.ps1, setting IsCustomDomainEnabled to $false for the new environment. The applications will still be available to access over https using the default host names provided by the Azure Container Apps service. However, the applications will not be accessible using the custom host names until the DNS and TLS configuration has been completed.

Tip

The DNS records and WAF rules can be registered even before the workload services are deployed to the new environment

  1. Register DNS records for the workload services that will be deployed to the new environment

    1. Follow the guidance Register DNS records to list the DNS records that need to be registered at the appropriate DNS zone
    2. Add the DNS records to the DNS zone for the new environment

  2. Add whitelists to Cloudflare Web Application Firewall (WAF)

    1. Follow the guidance Add whitelists to Cloudflare Web Application Firewall (WAF) to list the WAF rules that need to be registered at the appropriate WAF
    2. Add the WAF rules to the WAF for the new environment

  3. Review Cloudflare origin certs

    Cloudflare origin certs are used for secure communication between the cloudflare proxy and the workload services such as Azure Container Apps. These cloudflare origin certs typically will have been created as wildcard certs. As such a new subdomain for each workload service that will be deployed to the new environment should be covered by the existing wildcard cert.

    This should be confirmed by finding the details of the existing origin cert to check that the new host names are covered. For example:

    cloudflare origin cert

    If the existing origin cert does not cover the new subdomain, then you will need to create a new origin cert for the new hostname as explained in the section Add TLS certificates to shared key vault

5. Configure okta

  1. Configure okta (identity and token provider) for new environment

    As required, ensure the following okta configurations are created for the new environment:

    • Okta Authorization server with the same policies as the existing environments
    • Okta Application for each workload service that needs to acquire identity and/or access tokens

    Once created you will add in the following steps:

    • the client id and authorization server url as config values
    • the client secret as a keyvault secret

  2. Set okta variables in the app deployment github workflow

    Now that the okta configuration has been created, set the okta variables in __Application Deployment
    github workflow. For example, change the Map variables step to include the following variables:

    • Api_TokenProvider_Authority - the new okta authorization server url

6. Deploy infrastructure

Important

It might take several attempts to run the infrastructure github workflow for the new environment. That's ok, the infra-as-code deployment is idempotent, so if you run the workflow, and it fails, you can run it again, and it will only deploy the changes that are required.

  1. Deploy the infrastructure to the new environment

    Follow the guidance Deploying infrastructure from CI/CD to deploy the infrastructure for the new environment

    If you do need to run the workflow multiple times, you will need to:

    • Cancel the workflow run (yes, this is counterintuitive)
    • Click on the "Re-run failed jobs" button in the workflow
    • Approve the deployment to the new environment to kick off the deployment again

  2. Grant initial access to Azure environment

    To grant access to the new environment in Azure, follow the guidance Granting access to Azure or Power-BI resources

    You will need to grant yourself or someone who is responsible for assisting with the deployment, access to the new environment in Azure.

    That person will need to be able to add keyvault secrets which is the highest access rights available that can be granted to a user for the new environment. As described in the section Azure environment Access levels:

    • development team role (aka PD) has access to the keyvault deployed to dev/test environments
    • support-tier2 team role will have access to the keyvault deployed to all other environments

    To revoke access granted above, for example where access was granted to a developer to assist with the deployment, follow the guidance Revoking access to Azure resources

7. Deploy application code

Important

It will likely take 30 mins or so for permissions granted in previous steps above to take effect. You will not be able to access the keyvault nor other Azure resources in the new environment until the permissions have taken effect.

  1. Add any keyvault secrets for the new environment

    The infrastructure deployment will have created a new keyvault for the new environment. Review the existing secrets in use for the existing environments, and ensure that secret keys are added to the new keyvault as required.

    To find the names of the keyvaults in azure, you can use the following command:

    ./tools/infrastructure/print-product-convention-table.ps1 { $_.SubProducts.Keyvault } -AsArray | Select Env, ResourceName

  2. Deploy the application to the new environment

    Follow the guidance Deploying app from CI/CD to deploy the application code for the new environment

8. Test the application

  1. Add postman environment file for the new environment

    The postman environment files to create are in the folder postman.

    Create for the following postman environment files:

  2. Test the application in the new environment

    To find the host names of the workload services, run the following command:

    ./tools/infrastructure/print-product-convention-table.ps1 { $_.SubProducts.Values } -AsArray | 
  ? { $_.Type -in 'AksPod', 'AcaApp' } | Select Env, Name, HostName

    Test the workloads using the following postman collections: