From 0a151535fa2de33a588b58afc09cb2636e44176e Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 14:40:18 -0500 Subject: [PATCH 01/89] Add documentation for multiple AWS accounts --- docs/infra/set-up-aws-account.md | 4 --- docs/infra/set-up-aws-accounts.md | 53 +++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+), 4 deletions(-) create mode 100644 docs/infra/set-up-aws-accounts.md diff --git a/docs/infra/set-up-aws-account.md b/docs/infra/set-up-aws-account.md index 47af9dbcc..dd8495618 100644 --- a/docs/infra/set-up-aws-account.md +++ b/docs/infra/set-up-aws-account.md @@ -37,10 +37,6 @@ make infra-set-up-account ACCOUNT_NAME= This command will create the S3 tfstate bucket and the GitHub OIDC provider. It will also create a `[account name].[account id].s3.tfbackend` file in the `infra/accounts` directory. -### 3. Update the account names map in app-config - -In [app-config/main.tf](/infra/app/app-config/main.tf), update the `account_names_by_environment` config to reflect the account name you chose. - ## Making changes to the account If you make changes to the account terraform and want to apply those changes, run diff --git a/docs/infra/set-up-aws-accounts.md b/docs/infra/set-up-aws-accounts.md new file mode 100644 index 000000000..232dcea73 --- /dev/null +++ b/docs/infra/set-up-aws-accounts.md @@ -0,0 +1,53 @@ +# Set up AWS accounts + +The infrastructure supports managing resources across different [AWS accounts](https://docs.aws.amazon.com/accounts/latest/reference/accounts-welcome.html) by mapping different environments (e.g. `dev`, `staging`, `prod`) to specific AWS accounts. Shared resources, such as the build repository, can also be deployed into a specified AWS account. + +The environment to AWS account mapping is specified for each application deployed by this infrastructure. This means that multiple applications can share AWS accounts or be deployed into different AWS accounts. + +## Prerequisites + +* You'll need to have [set up infrastructure tools](./set-up-infrastructure-tools.md), like Terraform, AWS CLI, and AWS authentication. + +## Instructions + +### 1. Decide on AWS account strategy + +A simple project might have only one AWS account and all environments should be deployed to this environment. This mapping might look like this: + +```mermaid +flowchart TD + shared --> my_aws_account + dev --> my_aws_account + staging --> my_aws_account + prod --> my_aws_account +``` + +A more complex project might have separate AWS accounts for environment, enhancing security by isolating each environment into completely separate AWS accounts. This mapping might look like this: + +```mermaid +flowchart TD + shared --> shared_aws_account + dev --> dev_aws_account + staging --> staging_aws_account + prod --> prod_aws_account +``` + +A project could also isolate just the `prod` environment and group the lower environments: + +```mermaid +flowchart TD + shared --> shared_aws_account + dev --> lower_aws_account + staging --> lower_aws_account + prod --> prod_aws_account +``` + +Decide on the strategy that is appropriate for your project. + +### 2. Ensure AWS account(s) have been created + +For **each** AWS account you wish to use, ensure that you have created it in AWS and are able to authenticate to it. + +### 3. Set up AWS account + +For **each** AWS account you wish to use, follow the instructions in (Set Up AWS Account)[./set-up-aws-account.md]. \ No newline at end of file From 6931f1e082aa5622f05d9715033408f206a68c61 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 15:38:41 -0500 Subject: [PATCH 02/89] Reword for multiple AWS accounts --- docs/infra/set-up-aws-accounts.md | 4 ++-- infra/README.md | 19 +++++++++---------- 2 files changed, 11 insertions(+), 12 deletions(-) diff --git a/docs/infra/set-up-aws-accounts.md b/docs/infra/set-up-aws-accounts.md index 232dcea73..a28249315 100644 --- a/docs/infra/set-up-aws-accounts.md +++ b/docs/infra/set-up-aws-accounts.md @@ -46,8 +46,8 @@ Decide on the strategy that is appropriate for your project. ### 2. Ensure AWS account(s) have been created -For **each** AWS account you wish to use, ensure that you have created it in AWS and are able to authenticate to it. +For **each** AWS account you wish to use, ensure it has been created in AWS and you are able to authenticate to it. ### 3. Set up AWS account -For **each** AWS account you wish to use, follow the instructions in (Set Up AWS Account)[./set-up-aws-account.md]. \ No newline at end of file +For **each** AWS account you wish to use, follow the instructions in (set-up-aws-account)[./set-up-aws-account.md]. \ No newline at end of file diff --git a/infra/README.md b/infra/README.md index 57412383a..0acc52ff0 100644 --- a/infra/README.md +++ b/infra/README.md @@ -9,7 +9,7 @@ The structure for the infrastructure code looks like this: ```text infra/ Infrastructure code accounts/ [Root module] IaC and IAM resources - [app_name]/ Application directory: infrastructure for the main application + [APP_NAME]/ Application directory: infrastructure for the [APP_NAME] application modules/ Reusable child modules networks/ [Root module] Account level network config (shared across all apps, environments, and terraform workspaces) ``` @@ -45,7 +45,7 @@ This project has the following AWS environments: - `staging` - `prod` -The environments share the same root modules but will have different configurations. Backend configuration is saved as [`.tfbackend`](https://developer.hashicorp.com/terraform/language/settings/backends/configuration#file) files. Most `.tfbackend` files are named after the environment. For example, the `[app_name]/service` infrastructure resources for the `dev` environment are configured via `dev.s3.tfbackend`. Resources for a module that are shared across environments, such as the build-repository, use `shared.s3.tfbackend`. Resources that are shared across the entire account (e.g. /infra/accounts) use `..s3.tfbackend`. +The environments share the same root modules but will have different configurations. Backend configuration is saved as [`.tfbackend`](https://developer.hashicorp.com/terraform/language/settings/backends/configuration#file) files. Most `.tfbackend` files are named after the environment. For example, the `[APP_NAME]/service` infrastructure resources for the `dev` environment are configured via `dev.s3.tfbackend`. Resources for a module that are shared across environments, such as the build-repository, use `shared.s3.tfbackend`. Resources that are shared across the entire account (e.g. /infra/accounts) use `..s3.tfbackend`. ### 🔀 Project workflow @@ -57,15 +57,14 @@ Generally you should use the Make targets or the underlying bin scripts, but you ### 1️⃣ First time initialization - - -To set up this project for the first time (aka it has never been deployed to the target AWS account): +To set up this project for the first time, if it has never been deployed to the target AWS account(s): 1. [Install this template](/README.md#installation) into an application that meets the [Application Requirements](/README.md#application-requirements) -2. [Configure the project](/infra/project-config/main.tf) (These values will be used in subsequent infra setup steps to namespace resources and add infrastructure tags.) +2. [Configure the project](/infra/project-config/main.tf) (These values will be used in subsequent infra setup steps to namespace resources and add infrastructure tags) 3. [Set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md) -4. [Set up AWS account](/docs/infra/set-up-aws-account.md) -5. [Set up the virtual network (VPC)](/docs/infra/set-up-network.md) +4. [Set up AWS account(s)](/docs/infra/set-up-aws-accounts.md) +5. For each AWS account: + 1. [Set up virtual private network (VPC)](/docs/infra/set-up-network.md) 6. For each application: 1. [Set up application build repository](/docs/infra/set-up-app-build-repository.md) 2. [Set up application database](/docs/infra/set-up-database.md) @@ -75,9 +74,9 @@ To set up this project for the first time (aka it has never been deployed to the ### 🆕 New developer -To get set up as a new developer to a project that has already been deployed to the target AWS account: +To get set up as a new developer to the project, if it has already been deployed to the target AWS account(s): -1. [Set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md) +1. [Set up your infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md) 2. [Review how to make changes to infrastructure](/docs/infra/making-infra-changes.md) 3. (Optional) Set up a [terraform workspace](/docs/infra/intro-to-terraform-workspaces.md) From d58024fd68ffb8716f426aae4273617df8a3f895 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 15:42:24 -0500 Subject: [PATCH 03/89] Fix typos --- docs/infra/set-up-aws-accounts.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/infra/set-up-aws-accounts.md b/docs/infra/set-up-aws-accounts.md index a28249315..6f89489a8 100644 --- a/docs/infra/set-up-aws-accounts.md +++ b/docs/infra/set-up-aws-accounts.md @@ -1,8 +1,8 @@ # Set up AWS accounts -The infrastructure supports managing resources across different [AWS accounts](https://docs.aws.amazon.com/accounts/latest/reference/accounts-welcome.html) by mapping different environments (e.g. `dev`, `staging`, `prod`) to specific AWS accounts. Shared resources, such as the build repository, can also be deployed into a specified AWS account. +The infrastructure supports managing resources across different [AWS accounts](https://docs.aws.amazon.com/accounts/latest/reference/accounts-welcome.html) by mapping different environments (e.g. `dev`, `staging`, `prod`) to specified AWS accounts. Resources that are shared across environments, such as build repositories, are also deployed into a specified AWS account. -The environment to AWS account mapping is specified for each application deployed by this infrastructure. This means that multiple applications can share AWS accounts or be deployed into different AWS accounts. +The environment-to-AWS-account mapping is specified for each application deployed by this infrastructure. This means that multiple applications can share AWS accounts or be deployed into different AWS accounts. ## Prerequisites @@ -50,4 +50,4 @@ For **each** AWS account you wish to use, ensure it has been created in AWS and ### 3. Set up AWS account -For **each** AWS account you wish to use, follow the instructions in (set-up-aws-account)[./set-up-aws-account.md]. \ No newline at end of file +For **each** AWS account you wish to use, follow the instructions in [set-up-aws-account](./set-up-aws-account.md). \ No newline at end of file From 1dfb11592ee8e8369284cac9b6917677b889ea72 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 16:05:13 -0500 Subject: [PATCH 04/89] Add documentation for configuring apps --- docs/infra/set-up-app-config.md | 48 +++++++++++++++++++++++++++++++++ docs/infra/set-up-network.md | 6 +++-- infra/README.md | 2 ++ 3 files changed, 54 insertions(+), 2 deletions(-) create mode 100644 docs/infra/set-up-app-config.md diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md new file mode 100644 index 000000000..42b354c7c --- /dev/null +++ b/docs/infra/set-up-app-config.md @@ -0,0 +1,48 @@ +# Set up application config + +The application config setup process will configure the application. These values will be used in subsequent infra setup steps to determine which resources to deploy and how they will be configured. + +This setup process applies to each application in the project. + +## Requirements + +Before setting up the application's config you'll need to have: + +1. [Configure the project](/infra/project-config/main.tf) +2. [Set up the AWS account(s)](./set-up-aws-accounts.md) + +## 1. Edit the app-config + +Modify the following values in the application's `app-config`. For instance, in `/infra/app/app-config/main.tf`. + +## 2. Determine environments + +Set the `environments` array to list the names of the environments for this application. + +By default, this is set to `["dev", "staging", "prod"]`. + +## 3. Indicate whether or not the application has a database + +Set `has_database` to `true` or `false` to indicate whether a database is needed. + +By default, this is set to `false`. + +## 4. Indicate whether or not the application has external services + +Set `has_external_non_aws_service` to `true` or `false` to indicate whether the network should allow the application to access to resources outside of the VPC. + +By default, this is set to `false`. + +## 5. Indicate whether or not the application should integrate with an incident management service + +Set `has_incident_management_service` to `true` or `false` to indicate whether the application should integrate with an incident management service. + +By default, this is set to `false`. + +## 6. Configure the environments + +Set the `account_names_by_environment` hash to map environments to AWS accounts. See [set up AWS accounts](./set-up-aws-accounts.md) for more information. + +## Set up application build repository + +Once you set up the application config, you can proceed to [set up application build repository](./set-up-app-build-repository.md) \ No newline at end of file diff --git a/docs/infra/set-up-network.md b/docs/infra/set-up-network.md index bdd99638d..2986e80ad 100644 --- a/docs/infra/set-up-network.md +++ b/docs/infra/set-up-network.md @@ -6,12 +6,14 @@ The network setup process will configure and deploy network resources needed by 2. Create public subnets for publicly accessible resources such as the application load balancer, private subnets for the application service, and private subnets for the database. 3. Create VPC endpoints for the AWS services needed by ECS Fargate to fetch the container image and log to AWS CloudWatch. If your application has a database, it will also create VPC endpoints for the AWS services needed by the database layer and a security group to contain those VPC endpoints. +This setup process applies to each AWS account in the project. + ## Requirements Before setting up the network you'll need to have: -1. [Set up the AWS account](./set-up-aws-account.md) -2. Optionally adjust the configuration for the networks you want to have on your project in the [project-config module](/infra/project-config/networks.tf). By default there are three networks defined, one for each application environment. If you have multiple apps and want your applications in separate networks, you may want to give the networks differentiating names (e.g. "foo-dev", "foo-prod", "bar-dev", "bar-prod", instead of just "dev", "prod"). +1. [Set up the AWS account](./set-up-aws-account.md). If you have multiple AWS accounts in your project, see [set-up-aws-accounts](./set-up-aws-accounts.md). +2. Optionally adjust the configuration for the networks you want to have on your project in the [project-config module](/infra/project-config/networks.tf). By default there are three networks defined, one for each application environment. If you have multiple applications and want your applications in separate networks within the same AWS account, you may want to give the networks differentiating names (e.g. "foo-dev", "foo-prod", "bar-dev", "bar-prod", instead of just "dev", "prod"). 1. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md). You can also come back to setting up custom domains at a later time. 2. Optionally, [configure HTTPS support](/docs/infra/https-support.md). You can also come back to setting up HTTPS support at a later time. 3. [Configure the app](/infra/app/app-config/main.tf). diff --git a/infra/README.md b/infra/README.md index 0acc52ff0..3cfea86a7 100644 --- a/infra/README.md +++ b/infra/README.md @@ -63,6 +63,8 @@ To set up this project for the first time, if it has never been deployed to the 2. [Configure the project](/infra/project-config/main.tf) (These values will be used in subsequent infra setup steps to namespace resources and add infrastructure tags) 3. [Set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md) 4. [Set up AWS account(s)](/docs/infra/set-up-aws-accounts.md) +5. For each application: + 1. [Configure the application](/docs/infra/set-up-app-config.md) 5. For each AWS account: 1. [Set up virtual private network (VPC)](/docs/infra/set-up-network.md) 6. For each application: From c75a43dfea59014a762b1ca9577f6af88b325b3a Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 16:12:28 -0500 Subject: [PATCH 05/89] Make READMEs less circular --- infra/README.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/infra/README.md b/infra/README.md index 3cfea86a7..bc9344581 100644 --- a/infra/README.md +++ b/infra/README.md @@ -57,13 +57,20 @@ Generally you should use the Make targets or the underlying bin scripts, but you ### 1️⃣ First time initialization -To set up this project for the first time, if it has never been deployed to the target AWS account(s): +#### Requirements + +Before setting up the infrastructure, you'll need to: 1. [Install this template](/README.md#installation) into an application that meets the [Application Requirements](/README.md#application-requirements) -2. [Configure the project](/infra/project-config/main.tf) (These values will be used in subsequent infra setup steps to namespace resources and add infrastructure tags) -3. [Set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md) -4. [Set up AWS account(s)](/docs/infra/set-up-aws-accounts.md) -5. For each application: + +#### Instructions + +To set up this project for the first time, if it has never been deployed to the target AWS account(s): + +1. [Configure the project](/infra/project-config/main.tf) (These values will be used in subsequent infra setup steps to namespace resources and add infrastructure tags) +2. [Set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md) +3. [Set up AWS account(s)](/docs/infra/set-up-aws-accounts.md) +4. For each application: 1. [Configure the application](/docs/infra/set-up-app-config.md) 5. For each AWS account: 1. [Set up virtual private network (VPC)](/docs/infra/set-up-network.md) From f9fe7922f3a964ab648278079cf90c9afe81b452 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 16:12:44 -0500 Subject: [PATCH 06/89] Remove brittle next step --- docs/infra/set-up-app-config.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md index 42b354c7c..f95469b26 100644 --- a/docs/infra/set-up-app-config.md +++ b/docs/infra/set-up-app-config.md @@ -42,7 +42,3 @@ By default, this is set to `false`. ## 6. Configure the environments Set the `account_names_by_environment` hash to map environments to AWS accounts. See [set up AWS accounts](./set-up-aws-accounts.md) for more information. - -## Set up application build repository - -Once you set up the application config, you can proceed to [set up application build repository](./set-up-app-build-repository.md) \ No newline at end of file From 87bd8dd27d2352f3cb2519f7a7f0d10bda4ec641 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 16:18:17 -0500 Subject: [PATCH 07/89] Fix headings and use consistent prereq terminology --- docs/infra/set-up-app-config.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md index f95469b26..95d5a25d5 100644 --- a/docs/infra/set-up-app-config.md +++ b/docs/infra/set-up-app-config.md @@ -4,41 +4,41 @@ The application config setup process will configure the application. These value This setup process applies to each application in the project. -## Requirements +## Prerequisites -Before setting up the application's config you'll need to have: +1. You'll need to have [configured the project](/infra/project-config/main.tf) +2. You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) -1. [Configure the project](/infra/project-config/main.tf) -2. [Set up the AWS account(s)](./set-up-aws-accounts.md) +## Instructions -## 1. Edit the app-config +### 1. Edit the app-config Modify the following values in the application's `app-config`. For instance, in `/infra/app/app-config/main.tf`. -## 2. Determine environments +### 2. Determine environments Set the `environments` array to list the names of the environments for this application. By default, this is set to `["dev", "staging", "prod"]`. -## 3. Indicate whether or not the application has a database +### 3. Indicate whether or not the application has a database Set `has_database` to `true` or `false` to indicate whether a database is needed. By default, this is set to `false`. -## 4. Indicate whether or not the application has external services +### 4. Indicate whether or not the application has external services Set `has_external_non_aws_service` to `true` or `false` to indicate whether the network should allow the application to access to resources outside of the VPC. By default, this is set to `false`. -## 5. Indicate whether or not the application should integrate with an incident management service +### 5. Indicate whether or not the application should integrate with an incident management service Set `has_incident_management_service` to `true` or `false` to indicate whether the application should integrate with an incident management service. By default, this is set to `false`. -## 6. Configure the environments +### 6. Configure the environments Set the `account_names_by_environment` hash to map environments to AWS accounts. See [set up AWS accounts](./set-up-aws-accounts.md) for more information. From ce9ff9e8d7569966eaaa9c640ecd59060200006f Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 16:18:56 -0500 Subject: [PATCH 08/89] Use consistent prereq terminology --- infra/README.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/infra/README.md b/infra/README.md index bc9344581..7308fe29c 100644 --- a/infra/README.md +++ b/infra/README.md @@ -57,11 +57,9 @@ Generally you should use the Make targets or the underlying bin scripts, but you ### 1️⃣ First time initialization -#### Requirements +#### Prerequisites -Before setting up the infrastructure, you'll need to: - -1. [Install this template](/README.md#installation) into an application that meets the [Application Requirements](/README.md#application-requirements) +1. You'll need to have [installed this template](/README.md#installation) into an application that meets the [Application Requirements](/README.md#application-requirements) #### Instructions From 594dee04a99fe346254cd671b95490d120986f57 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 16:19:55 -0500 Subject: [PATCH 09/89] Fix link name --- docs/infra/set-up-aws-accounts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/infra/set-up-aws-accounts.md b/docs/infra/set-up-aws-accounts.md index 6f89489a8..2de53200b 100644 --- a/docs/infra/set-up-aws-accounts.md +++ b/docs/infra/set-up-aws-accounts.md @@ -50,4 +50,4 @@ For **each** AWS account you wish to use, ensure it has been created in AWS and ### 3. Set up AWS account -For **each** AWS account you wish to use, follow the instructions in [set-up-aws-account](./set-up-aws-account.md). \ No newline at end of file +For **each** AWS account you wish to use, [set up the AWS account](./set-up-aws-account.md). \ No newline at end of file From ba38e6a0fb8424d4bfe9db50d6c48820cb1ee0a0 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 16:21:19 -0500 Subject: [PATCH 10/89] Make markdown list notation consistent --- README.md | 6 +++--- docs/infra/set-up-app-config.md | 4 ++-- infra/README.md | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 19715d858..172c45db6 100644 --- a/README.md +++ b/README.md @@ -35,9 +35,9 @@ Now you're ready to set up the various pieces of your infrastructure. After downloading and installing the template into your project: 1. Follow the steps in [infra/README.md](/infra/README.md) to setup the infrastructure for your application. -1. After setting up AWS resources, you can [set up GitHub Actions workflows](./template-only-docs/set-up-ci.md). -1. After configuring GitHub Actions, you can [set up continuous deployment](./template-only-docs/set-up-cd.md). -1. At any point, [set up your team workflow](./template-only-docs/set-up-team-workflow.md). +2. After setting up AWS resources, you can [set up GitHub Actions workflows](./template-only-docs/set-up-ci.md). +3. After configuring GitHub Actions, you can [set up continuous deployment](./template-only-docs/set-up-cd.md). +4. At any point, [set up your team workflow](./template-only-docs/set-up-team-workflow.md). ## Updates diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md index 95d5a25d5..0c8d9daa5 100644 --- a/docs/infra/set-up-app-config.md +++ b/docs/infra/set-up-app-config.md @@ -6,8 +6,8 @@ This setup process applies to each application in the project. ## Prerequisites -1. You'll need to have [configured the project](/infra/project-config/main.tf) -2. You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) +* You'll need to have [configured the project](/infra/project-config/main.tf) +* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) ## Instructions diff --git a/infra/README.md b/infra/README.md index 7308fe29c..0bdeb4c7b 100644 --- a/infra/README.md +++ b/infra/README.md @@ -59,7 +59,7 @@ Generally you should use the Make targets or the underlying bin scripts, but you #### Prerequisites -1. You'll need to have [installed this template](/README.md#installation) into an application that meets the [Application Requirements](/README.md#application-requirements) +* You'll need to have [installed this template](/README.md#installation) into an application that meets the [Application Requirements](/README.md#application-requirements) #### Instructions From 392f5c34e35822dd67cc7d161584587bd02cf914 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 16:24:18 -0500 Subject: [PATCH 11/89] Add high level notes on what the doc covers --- docs/infra/set-up-aws-accounts.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/infra/set-up-aws-accounts.md b/docs/infra/set-up-aws-accounts.md index 2de53200b..99dd95c29 100644 --- a/docs/infra/set-up-aws-accounts.md +++ b/docs/infra/set-up-aws-accounts.md @@ -4,6 +4,11 @@ The infrastructure supports managing resources across different [AWS accounts](h The environment-to-AWS-account mapping is specified for each application deployed by this infrastructure. This means that multiple applications can share AWS accounts or be deployed into different AWS accounts. +The AWS account setup process will: + +1. Determine AWS account strategy +2. Configure all AWS accounts + ## Prerequisites * You'll need to have [set up infrastructure tools](./set-up-infrastructure-tools.md), like Terraform, AWS CLI, and AWS authentication. From fc5b686cd8db29fab8399dbdcc717ebb391919b2 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 16:37:38 -0500 Subject: [PATCH 12/89] Change instruction format --- docs/infra/set-up-app-config.md | 39 ++++++++++++--------------------- 1 file changed, 14 insertions(+), 25 deletions(-) diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md index 0c8d9daa5..d0041a80e 100644 --- a/docs/infra/set-up-app-config.md +++ b/docs/infra/set-up-app-config.md @@ -11,34 +11,23 @@ This setup process applies to each application in the project. ## Instructions -### 1. Edit the app-config +### 1. Configure the app-config -Modify the following values in the application's `app-config`. For instance, in `/infra/app/app-config/main.tf`. +Modify the following values in the application's `app-config` (e.g. in `/infra/app/app-config/main.tf`): -### 2. Determine environments +* Set the `environments` array to list the names of the environments for this application. By default, this is set to `["dev", "staging", "prod"]`. +* Set `has_database` to `true` or `false` to indicate whether or not the application has a database to integrate with. This setting determines whether or not to create VPC endpoints needed by the database layer. By default, this is set to `false`. +* Set `has_external_non_aws_service` to `true` or `false` to indicate whether or not your application makes calls to an external non-AWS service. This setting determines whether or not to create NAT gateways, which allows the service in the private subnet to make requests to the internet. By default, this is set to `false`. +* Set `has_incident_management_service` to `true` or `false` to indicate whether the application should integrate with an incident management service. By default, this is set to `false`. +* Set the `account_names_by_environment` hash to map environments to AWS accounts. See [set up AWS accounts](./set-up-aws-accounts.md) for more information. -Set the `environments` array to list the names of the environments for this application. +### 2. Configure each environment -By default, this is set to `["dev", "staging", "prod"]`. +Within the `app-config` directory (e.g. `infra/app/app-config`), each environment configured in the `environments` array in the previous step should have its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. -### 3. Indicate whether or not the application has a database +In each environment config file, modify the following values: -Set `has_database` to `true` or `false` to indicate whether a database is needed. - -By default, this is set to `false`. - -### 4. Indicate whether or not the application has external services - -Set `has_external_non_aws_service` to `true` or `false` to indicate whether the network should allow the application to access to resources outside of the VPC. - -By default, this is set to `false`. - -### 5. Indicate whether or not the application should integrate with an incident management service - -Set `has_incident_management_service` to `true` or `false` to indicate whether the application should integrate with an incident management service. - -By default, this is set to `false`. - -### 6. Configure the environments - -Set the `account_names_by_environment` hash to map environments to AWS accounts. See [set up AWS accounts](./set-up-aws-accounts.md) for more information. +* Set `environment` to the name of the environment. This should match the name of the file. +* Set `network_name`. By default, it should match the name of the environment. This mapping ensures that each network is configured appropriately based on the application(s) in that network (see `local.apps_in_network` in `/infra/networks/main.tf`). Failure to set the network name properly means that the network layer may not receive the correct application configurations for `has_database` and `has_external_non_aws_service`. +* Skip `domain_name` for now +* Skip `enable_https` for now From 42c1a9798760d16bee234a6e9523e395e30dca49 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 16:42:17 -0500 Subject: [PATCH 13/89] Modify network instructions --- docs/infra/set-up-network.md | 33 ++++++++++++++++++++++----------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/docs/infra/set-up-network.md b/docs/infra/set-up-network.md index 2986e80ad..a089e8c91 100644 --- a/docs/infra/set-up-network.md +++ b/docs/infra/set-up-network.md @@ -8,20 +8,31 @@ The network setup process will configure and deploy network resources needed by This setup process applies to each AWS account in the project. -## Requirements +## Prerequisites -Before setting up the network you'll need to have: - -1. [Set up the AWS account](./set-up-aws-account.md). If you have multiple AWS accounts in your project, see [set-up-aws-accounts](./set-up-aws-accounts.md). -2. Optionally adjust the configuration for the networks you want to have on your project in the [project-config module](/infra/project-config/networks.tf). By default there are three networks defined, one for each application environment. If you have multiple applications and want your applications in separate networks within the same AWS account, you may want to give the networks differentiating names (e.g. "foo-dev", "foo-prod", "bar-dev", "bar-prod", instead of just "dev", "prod"). +* You'll need to have [set up the AWS account](./set-up-aws-account.md). If you have multiple AWS accounts in your project, you'll need to have [set up all of the AWS accounts](./set-up-aws-accounts.md). +* Optionally, you'll need to have adjusted the configuration for the networks you want to have on your project in the [project-config module](/infra/project-config/networks.tf). By default there are three networks defined, one for each application environment. If you have multiple applications and want your applications in separate networks within the same AWS account, you may want to give the networks differentiating names (e.g. "foo-dev", "foo-prod", "bar-dev", "bar-prod", instead of just "dev", "prod"). 1. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md). You can also come back to setting up custom domains at a later time. 2. Optionally, [configure HTTPS support](/docs/infra/https-support.md). You can also come back to setting up HTTPS support at a later time. -3. [Configure the app](/infra/app/app-config/main.tf). - 1. Update `has_database` to `true` or `false` depending on whether or not your application has a database to integrate with. This setting determines whether or not to create VPC endpoints needed by the database layer. - 2. Update `has_external_non_aws_service` to `true` or `false` depending on whether or not your application makes calls to an external non-AWS service. This setting determines whether or not to create NAT gateways, which allows the service in the private subnet to make requests to the internet. - 3. Update `network_name` for your application environments. This mapping ensures that each network is configured appropriately based on the application(s) in that network (see `local.apps_in_network` in [/infra/networks/main.tf](/infra/networks/main.tf)) Failure to set the network name properly means that the network layer may not receive the correct application configurations for `has_database` and `has_external_non_aws_service`. +* You'll need to have configured [all applications](./set-up-app-config.md) + +## Instructions + +### 1. Make sure you're authenticated into the AWS account you want to configure + +The network is set up for whatever account you're authenticated into. To see which account that is, run + +```bash +aws sts get-caller-identity +``` + +To see a more human readable account alias instead of the account, run + +```bash +aws iam list-account-aliases +``` -## 1. Configure backend +### 2. Configure backend To create the tfbackend file for the new network, run @@ -29,7 +40,7 @@ To create the tfbackend file for the new network, run make infra-configure-network NETWORK_NAME= ``` -## 2. Create network resources +### 3. Create network resources Now run the following commands to create the resources. Review the terraform before confirming "yes" to apply the changes. From 352193c518e81ff7baafeb5db641ac32c32f2e0c Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 17:25:19 -0500 Subject: [PATCH 14/89] Start working on network docs --- docs/infra/set-up-aws-accounts.md | 21 ++++++++++++++++----- docs/infra/set-up-custom-domains.md | 18 ++++++++++-------- docs/infra/set-up-network.md | 2 -- infra/README.md | 2 ++ 4 files changed, 28 insertions(+), 15 deletions(-) diff --git a/docs/infra/set-up-aws-accounts.md b/docs/infra/set-up-aws-accounts.md index 99dd95c29..24c1325a3 100644 --- a/docs/infra/set-up-aws-accounts.md +++ b/docs/infra/set-up-aws-accounts.md @@ -6,8 +6,9 @@ The environment-to-AWS-account mapping is specified for each application deploye The AWS account setup process will: -1. Determine AWS account strategy -2. Configure all AWS accounts +1. Help you decide on your environments +2. Help you decide on your AWS account strategy +3. Configure all AWS accounts ## Prerequisites @@ -15,7 +16,17 @@ The AWS account setup process will: ## Instructions -### 1. Decide on AWS account strategy +### 1. Decide on environments + +Even though environment-to-AWS-account mapping is specified per-application. The names of the environments must be consistent across applications. Decide now what you would like the environments in your project to be. By default, the environments are: + +* dev +* staging +* prod + +You can always add new environments or delete existing environments later. + +### 2. Decide on AWS account strategy A simple project might have only one AWS account and all environments should be deployed to this environment. This mapping might look like this: @@ -49,10 +60,10 @@ flowchart TD Decide on the strategy that is appropriate for your project. -### 2. Ensure AWS account(s) have been created +### 3. Ensure AWS account(s) have been created For **each** AWS account you wish to use, ensure it has been created in AWS and you are able to authenticate to it. -### 3. Set up AWS account +### 4. Set up AWS account For **each** AWS account you wish to use, [set up the AWS account](./set-up-aws-account.md). \ No newline at end of file diff --git a/docs/infra/set-up-custom-domains.md b/docs/infra/set-up-custom-domains.md index e4ff84740..727658dd2 100644 --- a/docs/infra/set-up-custom-domains.md +++ b/docs/infra/set-up-custom-domains.md @@ -5,15 +5,17 @@ Production systems will want to set up custom domains to route internet traffic 1. Create an [Amazon Route 53 hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) to manage DNS records for a domain and subdomains 2. Create a DNS A (address) records to route traffic from a custom domain to the application's load balancer -## Requirements +## Prerequisites -Before setting up custom domains you'll need to have [set up the AWS account](./set-up-aws-account.md) +* You'll need to have [set up the AWS account](./set-up-aws-account.md). If you have multiple AWS accounts in your project, you'll need to have [set up all of the AWS accounts](./set-up-aws-accounts.md). -## 1. Set hosted zone in domain configuration +## Instructions + +### 1. Set hosted zone in domain configuration Update the value for the `hosted_zone` in the domain configuration. The custom domain configuration is defined as a `domain_config` object in the [network section of the project config module](/infra/project-config/networks.tf). A [hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) represents a domain and all of its subdomains. For example, a hosted zone of `platform-test.navateam.com` includes `platform-test.navateam.com`, `cdn.platform-test.navateam.com`, `notifications.platform-test.navateam.com`, `foo.bar.platform-test.navateam.com`, etc. -## 2. Update the network layer to create the hosted zone +### 2. Update the network layer to create the hosted zone Run the following command to create the hosted zone specified in the domain configuration. @@ -21,7 +23,7 @@ Run the following command to create the hosted zone specified in the domain conf make infra-update-network NETWORK_NAME= ``` -## 3. Delegate DNS requests to the newly created hosted zone +### 3. Delegate DNS requests to the newly created hosted zone You most likely registered your domain outside of this project. Using whichever service you used to register the domain name (e.g. Namecheap, GoDaddy, Google Domains, etc.), add a DNS NS (nameserver) record. Set the "name" equal to the `hosted_zone` and set the value equal to the list of hosted zone name servers that was created in the previous step. You can see the list of servers by running @@ -52,11 +54,11 @@ Run the following command to verify that DNS requests are being served by the ho nslookup -type=NS ``` -## 4. Configure custom domain for your application +### 4. Configure custom domain for your application Define the `domain_name` for each of the application environments in the `app-config` module. The `domain_name` must be either the same as the `hosted_zone` or a subdomain of the `hosted_zone`. For example, if your hosted zone is `platform-test.navateam.com`, then `platform-test.navateam.com` and `cdn.platform-test.navateam.com` are both valid values for `domain_name`. -## 5. Create A (address) records to route traffice from the custom domain to your application's load balancer +### 5. Create A (address) records to route traffice from the custom domain to your application's load balancer Run the following command to create the A record that routes traffic from the custom domain to the application's load balancer. @@ -64,7 +66,7 @@ Run the following command to create the A record that routes traffic from the cu make infra-update-app-service APP_NAME= ENVIRONMENT= ``` -## 6. Repeat for each application +### 6. Repeat for each application If you have multiple applications in the same network, repeat steps 4 and 5 for each application. diff --git a/docs/infra/set-up-network.md b/docs/infra/set-up-network.md index a089e8c91..89503c82a 100644 --- a/docs/infra/set-up-network.md +++ b/docs/infra/set-up-network.md @@ -12,8 +12,6 @@ This setup process applies to each AWS account in the project. * You'll need to have [set up the AWS account](./set-up-aws-account.md). If you have multiple AWS accounts in your project, you'll need to have [set up all of the AWS accounts](./set-up-aws-accounts.md). * Optionally, you'll need to have adjusted the configuration for the networks you want to have on your project in the [project-config module](/infra/project-config/networks.tf). By default there are three networks defined, one for each application environment. If you have multiple applications and want your applications in separate networks within the same AWS account, you may want to give the networks differentiating names (e.g. "foo-dev", "foo-prod", "bar-dev", "bar-prod", instead of just "dev", "prod"). - 1. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md). You can also come back to setting up custom domains at a later time. - 2. Optionally, [configure HTTPS support](/docs/infra/https-support.md). You can also come back to setting up HTTPS support at a later time. * You'll need to have configured [all applications](./set-up-app-config.md) ## Instructions diff --git a/infra/README.md b/infra/README.md index 0bdeb4c7b..df28ab422 100644 --- a/infra/README.md +++ b/infra/README.md @@ -72,6 +72,8 @@ To set up this project for the first time, if it has never been deployed to the 1. [Configure the application](/docs/infra/set-up-app-config.md) 5. For each AWS account: 1. [Set up virtual private network (VPC)](/docs/infra/set-up-network.md) + 1. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md). You can also come back to setting up custom domains at a later time. + 2. Optionally, [configure HTTPS support](/docs/infra/https-support.md). You can also come back to setting up HTTPS support at a later time. 6. For each application: 1. [Set up application build repository](/docs/infra/set-up-app-build-repository.md) 2. [Set up application database](/docs/infra/set-up-database.md) From e329a982ea988191e17722d1a66f1bb83a1b764b Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 17:39:38 -0500 Subject: [PATCH 15/89] Modify network docs. Make custom domains actually optional. --- docs/infra/set-up-aws-accounts.md | 2 +- docs/infra/set-up-network.md | 19 ++++++++++++++----- infra/networks/main.tf | 13 +++++++------ 3 files changed, 22 insertions(+), 12 deletions(-) diff --git a/docs/infra/set-up-aws-accounts.md b/docs/infra/set-up-aws-accounts.md index 24c1325a3..108e6bedd 100644 --- a/docs/infra/set-up-aws-accounts.md +++ b/docs/infra/set-up-aws-accounts.md @@ -18,7 +18,7 @@ The AWS account setup process will: ### 1. Decide on environments -Even though environment-to-AWS-account mapping is specified per-application. The names of the environments must be consistent across applications. Decide now what you would like the environments in your project to be. By default, the environments are: +Even though environment-to-AWS-account mapping is specified per-application, the names of the environments must be consistent across applications. Decide now what you would like the environments in your project to be. By default, the environments are: * dev * staging diff --git a/docs/infra/set-up-network.md b/docs/infra/set-up-network.md index 89503c82a..7c35fa714 100644 --- a/docs/infra/set-up-network.md +++ b/docs/infra/set-up-network.md @@ -10,9 +10,8 @@ This setup process applies to each AWS account in the project. ## Prerequisites -* You'll need to have [set up the AWS account](./set-up-aws-account.md). If you have multiple AWS accounts in your project, you'll need to have [set up all of the AWS accounts](./set-up-aws-accounts.md). -* Optionally, you'll need to have adjusted the configuration for the networks you want to have on your project in the [project-config module](/infra/project-config/networks.tf). By default there are three networks defined, one for each application environment. If you have multiple applications and want your applications in separate networks within the same AWS account, you may want to give the networks differentiating names (e.g. "foo-dev", "foo-prod", "bar-dev", "bar-prod", instead of just "dev", "prod"). -* You'll need to have configured [all applications](./set-up-app-config.md) +* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You'll need to have configured [all applications](./set-up-app-config.md). ## Instructions @@ -30,7 +29,17 @@ To see a more human readable account alias instead of the account, run aws iam list-account-aliases ``` -### 2. Configure backend +### 2. Configure the project's network + +Modify the [project-config module](/infra/project-config/networks.tf) to ensure the environments match what you decided in the [set up AWS accounts](./set-up-aws-accounts.md) step. + +By default there are three networks defined, one for each application environment. You can add additional additional networks as desired. + +If you have multiple applications and want your applications in separate networks within the same AWS account, you may want to give the networks differentiating names (e.g. "foo-dev", "foo-prod", "bar-dev", "bar-prod", instead of just "dev", "prod"). + +Skip the `domain_config` config for now. This is addressed in [setting up custom domains](./set-up-custom-domains.md). + +### 3. Configure backend To create the tfbackend file for the new network, run @@ -38,7 +47,7 @@ To create the tfbackend file for the new network, run make infra-configure-network NETWORK_NAME= ``` -### 3. Create network resources +### 4. Create network resources Now run the following commands to create the resources. Review the terraform before confirming "yes" to apply the changes. diff --git a/infra/networks/main.tf b/infra/networks/main.tf index 554f0decb..3e615d1ab 100644 --- a/infra/networks/main.tf +++ b/infra/networks/main.tf @@ -69,9 +69,10 @@ module "network" { has_external_non_aws_service = local.has_external_non_aws_service } -module "domain" { - source = "../modules/domain" - name = local.domain_config.hosted_zone - manage_dns = local.domain_config.manage_dns - certificate_configs = local.domain_config.certificate_configs -} +# Uncomment the following section to enable custom domain management. +# module "domain" { +# source = "../modules/domain" +# name = local.domain_config.hosted_zone +# manage_dns = local.domain_config.manage_dns +# certificate_configs = local.domain_config.certificate_configs +# } From 83ceda9052b32da1c0be7e23f11341244b6ee001 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 18:07:01 -0500 Subject: [PATCH 16/89] WIP --- docs/infra/set-up-app-build-repository.md | 14 +++++------- docs/infra/set-up-aws-account.md | 1 + docs/infra/set-up-custom-domains.md | 25 ++++++++++++++++---- docs/infra/set-up-network.md | 21 ++++------------- docs/infra/set-up-networks.md | 28 +++++++++++++++++++++++ infra/README.md | 6 ++--- 6 files changed, 63 insertions(+), 32 deletions(-) create mode 100644 docs/infra/set-up-networks.md diff --git a/docs/infra/set-up-app-build-repository.md b/docs/infra/set-up-app-build-repository.md index 2f91ec211..48e633370 100644 --- a/docs/infra/set-up-app-build-repository.md +++ b/docs/infra/set-up-app-build-repository.md @@ -2,11 +2,13 @@ The application build repository setup process will create infrastructure resources needed to store built release candidate artifacts used to deploy the application to an environment. +This setup process applies to each application in the project. + ## Requirements Before setting up the application's build repository you'll need to have: -1. [Set up the AWS account](./set-up-aws-account.md) +1. [Set up the AWS account(s)](./set-up-aws-accounts.md) 2. [Configure the app](/infra/app/app-config/main.tf) ## 1. Configure backend @@ -14,7 +16,7 @@ Before setting up the application's build repository you'll need to have: To create the tfbackend file for the build repository using the backend configuration values from your current AWS account, run ```bash -make infra-configure-app-build-repository APP_NAME=app +make infra-configure-app-build-repository APP_NAME= ``` Pass in the name of the app folder within `infra`. By default this is `app`. @@ -24,9 +26,5 @@ Pass in the name of the app folder within `infra`. By default this is `app`. Now run the following commands to create the resources, making sure to verify the plan before confirming the apply. ```bash -make infra-update-app-build-repository APP_NAME=app -``` - -## Set up application environments - -Once you set up the deployment process, you can proceed to [set up application environments](./set-up-app-env.md) +make infra-update-app-build-repository APP_NAME= +``` \ No newline at end of file diff --git a/docs/infra/set-up-aws-account.md b/docs/infra/set-up-aws-account.md index dd8495618..36dbb4e44 100644 --- a/docs/infra/set-up-aws-account.md +++ b/docs/infra/set-up-aws-account.md @@ -10,6 +10,7 @@ The AWS account setup process will: * You'll need to have [set up infrastructure tools](./set-up-infrastructure-tools.md), like Terraform, AWS CLI, and AWS authentication. * You'll also need to make sure the [project is configured](/infra/project-config/main.tf). +* You'll need to have [decided on your environments and AWS account strategy](./set-up-aws-accounts.md) ## Instructions diff --git a/docs/infra/set-up-custom-domains.md b/docs/infra/set-up-custom-domains.md index 727658dd2..c31eef547 100644 --- a/docs/infra/set-up-custom-domains.md +++ b/docs/infra/set-up-custom-domains.md @@ -3,19 +3,32 @@ Production systems will want to set up custom domains to route internet traffic to their application services rather than using AWS-generated hostnames for the load balancers or the CDN. This document describes how to configure custom domains. The custom domain setup process will: 1. Create an [Amazon Route 53 hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) to manage DNS records for a domain and subdomains -2. Create a DNS A (address) records to route traffic from a custom domain to the application's load balancer +2. Create a DNS A (address) records to route traffic from a custom domain to an application's load balancer ## Prerequisites -* You'll need to have [set up the AWS account](./set-up-aws-account.md). If you have multiple AWS accounts in your project, you'll need to have [set up all of the AWS accounts](./set-up-aws-accounts.md). +* You'll need to have registered custom domain(s) with a domain registrar (e.g. Namecheap, GoDaddy, Google Domains, etc.). +* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You'll need to have [configured all applications](./set-up-app-config.md). +* You'll need to have [set up the networks](./set-up-network.md) that you want to add the custom domain to. ## Instructions ### 1. Set hosted zone in domain configuration -Update the value for the `hosted_zone` in the domain configuration. The custom domain configuration is defined as a `domain_config` object in the [network section of the project config module](/infra/project-config/networks.tf). A [hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) represents a domain and all of its subdomains. For example, a hosted zone of `platform-test.navateam.com` includes `platform-test.navateam.com`, `cdn.platform-test.navateam.com`, `notifications.platform-test.navateam.com`, `foo.bar.platform-test.navateam.com`, etc. +The custom domain configuration is defined as a `domain_config` object in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf). A [hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) represents a domain and all of its subdomains. -### 2. Update the network layer to create the hosted zone +Modify the `hosted_zone` [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) value for each network to match the custom domain (or a subdomain of the custom domain) that you registered. Each `hosted_zone` value must be different. + +For example, a hosted zone of `platform-test.navateam.com` includes `platform-test.navateam.com`, `cdn.platform-test.navateam.com`, `notifications.platform-test.navateam.com`, `foo.bar.platform-test.navateam.com`, etc. + +### 2. Enable custom domains + +In the [network module](/infra/networks/main.tf), enable custom domains by uncommenting the `module "domain"` section. + +---- + +### 3. Update the network layer to create the hosted zones Run the following command to create the hosted zone specified in the domain configuration. @@ -58,7 +71,9 @@ nslookup -type=NS Define the `domain_name` for each of the application environments in the `app-config` module. The `domain_name` must be either the same as the `hosted_zone` or a subdomain of the `hosted_zone`. For example, if your hosted zone is `platform-test.navateam.com`, then `platform-test.navateam.com` and `cdn.platform-test.navateam.com` are both valid values for `domain_name`. -### 5. Create A (address) records to route traffice from the custom domain to your application's load balancer +### 5. Create DNS A (address) records to route traffic from the custom domain to your application's load balancer + +If created after.... Run the following command to create the A record that routes traffic from the custom domain to the application's load balancer. diff --git a/docs/infra/set-up-network.md b/docs/infra/set-up-network.md index 7c35fa714..77e768e47 100644 --- a/docs/infra/set-up-network.md +++ b/docs/infra/set-up-network.md @@ -1,17 +1,16 @@ # Set up network -The network setup process will configure and deploy network resources needed by other modules. In particular, it will: +The network setup process will configure and deploy network resources needed by other modules for **one network**. In particular, it will: 1. Create a nondefault VPC 2. Create public subnets for publicly accessible resources such as the application load balancer, private subnets for the application service, and private subnets for the database. 3. Create VPC endpoints for the AWS services needed by ECS Fargate to fetch the container image and log to AWS CloudWatch. If your application has a database, it will also create VPC endpoints for the AWS services needed by the database layer and a security group to contain those VPC endpoints. -This setup process applies to each AWS account in the project. - ## Prerequisites * You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md). -* You'll need to have configured [all applications](./set-up-app-config.md). +* You'll need to have [configured all applications](./set-up-app-config.md). +* You'll need to have [configured the project's networks](./set-up-networks.md). ## Instructions @@ -29,17 +28,7 @@ To see a more human readable account alias instead of the account, run aws iam list-account-aliases ``` -### 2. Configure the project's network - -Modify the [project-config module](/infra/project-config/networks.tf) to ensure the environments match what you decided in the [set up AWS accounts](./set-up-aws-accounts.md) step. - -By default there are three networks defined, one for each application environment. You can add additional additional networks as desired. - -If you have multiple applications and want your applications in separate networks within the same AWS account, you may want to give the networks differentiating names (e.g. "foo-dev", "foo-prod", "bar-dev", "bar-prod", instead of just "dev", "prod"). - -Skip the `domain_config` config for now. This is addressed in [setting up custom domains](./set-up-custom-domains.md). - -### 3. Configure backend +### 2. Configure backend To create the tfbackend file for the new network, run @@ -47,7 +36,7 @@ To create the tfbackend file for the new network, run make infra-configure-network NETWORK_NAME= ``` -### 4. Create network resources +### 3. Create network resources Now run the following commands to create the resources. Review the terraform before confirming "yes" to apply the changes. diff --git a/docs/infra/set-up-networks.md b/docs/infra/set-up-networks.md new file mode 100644 index 000000000..8136c7fcd --- /dev/null +++ b/docs/infra/set-up-networks.md @@ -0,0 +1,28 @@ +# Set up network + +The network setup process will configure and deploy network resources needed by other modules. In particular, for each AWS account, it will: + +1. Create a nondefault VPC +2. Create public subnets for publicly accessible resources such as the application load balancer, private subnets for the application service, and private subnets for the database. +3. Create VPC endpoints for the AWS services needed by ECS Fargate to fetch the container image and log to AWS CloudWatch. If your application has a database, it will also create VPC endpoints for the AWS services needed by the database layer and a security group to contain those VPC endpoints. + +## Prerequisites + +* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You'll need to have [configured all applications](./set-up-app-config.md). + +## Instructions + +### 1. Configure the project's networks + +Modify [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) to ensure the environments listed match what you decided in the [set up AWS accounts](./set-up-aws-accounts.md) step. + +By default there are three networks defined, one for each application environment. You can add additional additional networks as desired. + +If you have multiple applications and want your applications in separate networks within the same AWS account, you may want to give the networks differentiating names (e.g. "foo-dev", "foo-prod", "bar-dev", "bar-prod", instead of just "dev", "prod"). + +Skip the `domain_config` config for now. This is addressed later in [setting up custom domains](./set-up-custom-domains.md). + +### 2. Set up each network + +For **each network** listed in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf), [set up the network](./set-up-network.md). \ No newline at end of file diff --git a/infra/README.md b/infra/README.md index df28ab422..24379558f 100644 --- a/infra/README.md +++ b/infra/README.md @@ -71,9 +71,9 @@ To set up this project for the first time, if it has never been deployed to the 4. For each application: 1. [Configure the application](/docs/infra/set-up-app-config.md) 5. For each AWS account: - 1. [Set up virtual private network (VPC)](/docs/infra/set-up-network.md) - 1. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md). You can also come back to setting up custom domains at a later time. - 2. Optionally, [configure HTTPS support](/docs/infra/https-support.md). You can also come back to setting up HTTPS support at a later time. + 1. [Set up the networks](/docs/infra/set-up-networks.md) + 2. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md). You can also come back to setting up custom domains at a later time. + 3. Optionally, [configure HTTPS support](/docs/infra/https-support.md). You can also come back to setting up HTTPS support at a later time. 6. For each application: 1. [Set up application build repository](/docs/infra/set-up-app-build-repository.md) 2. [Set up application database](/docs/infra/set-up-database.md) From 705dacfca6c900631df97c932fd4840f3ae3e579 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 25 Apr 2024 18:10:33 -0500 Subject: [PATCH 17/89] Wordsmith --- README.md | 2 +- docs/infra/set-up-custom-domains.md | 4 ++-- docs/infra/set-up-networks.md | 2 +- infra/README.md | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 172c45db6..c5cabbaa1 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ This template includes setup for: * **CD / Deployments** - infrastructure for continuous deployment, including: AWS account access for Github actions, scripts for building and publishing release artifacts, and a Github action for automated deployments from the main branch. * **Documentation** - technical documentation for the decisions that went into all the defaults that come with the template. -The system architecture will look like this (see [system architecture documentation](/docs/system-architecture.md) for more information): +By default, the system architecture will look like this (see [system architecture documentation](/docs/system-architecture.md) for more information): ![System architecture](https://lucid.app/publicSegments/view/e5a36152-200d-4d95-888e-4cdbdab80d1b/image.png) ## Application Requirements diff --git a/docs/infra/set-up-custom-domains.md b/docs/infra/set-up-custom-domains.md index c31eef547..f10cb7823 100644 --- a/docs/infra/set-up-custom-domains.md +++ b/docs/infra/set-up-custom-domains.md @@ -18,10 +18,10 @@ Production systems will want to set up custom domains to route internet traffic The custom domain configuration is defined as a `domain_config` object in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf). A [hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) represents a domain and all of its subdomains. -Modify the `hosted_zone` [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) value for each network to match the custom domain (or a subdomain of the custom domain) that you registered. Each `hosted_zone` value must be different. - For example, a hosted zone of `platform-test.navateam.com` includes `platform-test.navateam.com`, `cdn.platform-test.navateam.com`, `notifications.platform-test.navateam.com`, `foo.bar.platform-test.navateam.com`, etc. +For each network, modify the `hosted_zone` value in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) to match the custom domain (or a subdomain of the custom domain) that you registered. Each `hosted_zone` value must be different. + ### 2. Enable custom domains In the [network module](/infra/networks/main.tf), enable custom domains by uncommenting the `module "domain"` section. diff --git a/docs/infra/set-up-networks.md b/docs/infra/set-up-networks.md index 8136c7fcd..8ec118942 100644 --- a/docs/infra/set-up-networks.md +++ b/docs/infra/set-up-networks.md @@ -1,4 +1,4 @@ -# Set up network +# Set up networks The network setup process will configure and deploy network resources needed by other modules. In particular, for each AWS account, it will: diff --git a/infra/README.md b/infra/README.md index 24379558f..d77cf717b 100644 --- a/infra/README.md +++ b/infra/README.md @@ -71,7 +71,7 @@ To set up this project for the first time, if it has never been deployed to the 4. For each application: 1. [Configure the application](/docs/infra/set-up-app-config.md) 5. For each AWS account: - 1. [Set up the networks](/docs/infra/set-up-networks.md) + 1. [Set up the network(s)](/docs/infra/set-up-networks.md) 2. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md). You can also come back to setting up custom domains at a later time. 3. Optionally, [configure HTTPS support](/docs/infra/https-support.md). You can also come back to setting up HTTPS support at a later time. 6. For each application: From 28286fa3d891b4a2310578451cb0dfa10e26867a Mon Sep 17 00:00:00 2001 From: Rocket Date: Fri, 26 Apr 2024 14:18:31 -0500 Subject: [PATCH 18/89] WIP --- docs/infra/set-up-custom-domains.md | 12 +++++++----- docs/infra/set-up-networks.md | 4 ++-- infra/README.md | 7 +++---- 3 files changed, 12 insertions(+), 11 deletions(-) diff --git a/docs/infra/set-up-custom-domains.md b/docs/infra/set-up-custom-domains.md index f10cb7823..cfaac37e2 100644 --- a/docs/infra/set-up-custom-domains.md +++ b/docs/infra/set-up-custom-domains.md @@ -1,6 +1,8 @@ # Custom domains -Production systems will want to set up custom domains to route internet traffic to their application services rather than using AWS-generated hostnames for the load balancers or the CDN. This document describes how to configure custom domains. The custom domain setup process will: +Production systems will want to set up custom domains to route internet traffic to their application services rather than using AWS-generated hostnames for the load balancers or the CDN. This document describes how to configure custom domains. + +**For each network**, the custom domain setup process will: 1. Create an [Amazon Route 53 hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) to manage DNS records for a domain and subdomains 2. Create a DNS A (address) records to route traffic from a custom domain to an application's load balancer @@ -14,23 +16,23 @@ Production systems will want to set up custom domains to route internet traffic ## Instructions +Follow these instructions for **each network** you want to setup custom domains for. + ### 1. Set hosted zone in domain configuration The custom domain configuration is defined as a `domain_config` object in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf). A [hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) represents a domain and all of its subdomains. For example, a hosted zone of `platform-test.navateam.com` includes `platform-test.navateam.com`, `cdn.platform-test.navateam.com`, `notifications.platform-test.navateam.com`, `foo.bar.platform-test.navateam.com`, etc. -For each network, modify the `hosted_zone` value in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) to match the custom domain (or a subdomain of the custom domain) that you registered. Each `hosted_zone` value must be different. +**For each network** you want to use a custom domain, modify the `hosted_zone` value in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) to match the custom domain (or a subdomain of the custom domain) that you registered. Each `hosted_zone` value must be different. ### 2. Enable custom domains In the [network module](/infra/networks/main.tf), enable custom domains by uncommenting the `module "domain"` section. ----- - ### 3. Update the network layer to create the hosted zones -Run the following command to create the hosted zone specified in the domain configuration. +**For each network** you that you added a custom domain to in Step 1, run the following command to create the hosted zone specified in the domain configuration: ```bash make infra-update-network NETWORK_NAME= diff --git a/docs/infra/set-up-networks.md b/docs/infra/set-up-networks.md index 8ec118942..2be01de5d 100644 --- a/docs/infra/set-up-networks.md +++ b/docs/infra/set-up-networks.md @@ -21,8 +21,8 @@ By default there are three networks defined, one for each application environmen If you have multiple applications and want your applications in separate networks within the same AWS account, you may want to give the networks differentiating names (e.g. "foo-dev", "foo-prod", "bar-dev", "bar-prod", instead of just "dev", "prod"). -Skip the `domain_config` config for now. This is addressed later in [setting up custom domains](./set-up-custom-domains.md). +Skip the `domain_config` config for now. This is optionally configured later when [setting up custom domains](./set-up-custom-domains.md). ### 2. Set up each network -For **each network** listed in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf), [set up the network](./set-up-network.md). \ No newline at end of file +For **each network** listed in `/infra/project-config/networks.tf`, [set up the network](./set-up-network.md). \ No newline at end of file diff --git a/infra/README.md b/infra/README.md index d77cf717b..a2d03c778 100644 --- a/infra/README.md +++ b/infra/README.md @@ -70,16 +70,15 @@ To set up this project for the first time, if it has never been deployed to the 3. [Set up AWS account(s)](/docs/infra/set-up-aws-accounts.md) 4. For each application: 1. [Configure the application](/docs/infra/set-up-app-config.md) -5. For each AWS account: - 1. [Set up the network(s)](/docs/infra/set-up-networks.md) - 2. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md). You can also come back to setting up custom domains at a later time. - 3. Optionally, [configure HTTPS support](/docs/infra/https-support.md). You can also come back to setting up HTTPS support at a later time. +5. [Set up the network(s)](/docs/infra/set-up-networks.md) 6. For each application: 1. [Set up application build repository](/docs/infra/set-up-app-build-repository.md) 2. [Set up application database](/docs/infra/set-up-database.md) 3. [Set up application environment](/docs/infra/set-up-app-env.md) 4. [Configure environment variables and secrets](/docs/infra/environment-variables-and-secrets.md) 5. [Set up background jobs](/docs/infra/background-jobs.md) + 6. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md). You can also come back to setting up custom domains at a later time. + 7. Optionally, [configure HTTPS support](/docs/infra/https-support.md). You can also come back to setting up HTTPS support at a later time. ### 🆕 New developer From 6210dac303b30ac589d41e151f824c5a2b3f9b90 Mon Sep 17 00:00:00 2001 From: Rocket Date: Fri, 26 Apr 2024 14:21:55 -0500 Subject: [PATCH 19/89] Fix set-up-networks plural --- docs/infra/set-up-networks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/infra/set-up-networks.md b/docs/infra/set-up-networks.md index 2be01de5d..e5e54ed46 100644 --- a/docs/infra/set-up-networks.md +++ b/docs/infra/set-up-networks.md @@ -1,6 +1,6 @@ # Set up networks -The network setup process will configure and deploy network resources needed by other modules. In particular, for each AWS account, it will: +The network setup process will configure and deploy network resources needed by other modules. In particular, for each network, it will: 1. Create a nondefault VPC 2. Create public subnets for publicly accessible resources such as the application load balancer, private subnets for the application service, and private subnets for the database. @@ -21,7 +21,7 @@ By default there are three networks defined, one for each application environmen If you have multiple applications and want your applications in separate networks within the same AWS account, you may want to give the networks differentiating names (e.g. "foo-dev", "foo-prod", "bar-dev", "bar-prod", instead of just "dev", "prod"). -Skip the `domain_config` config for now. This is optionally configured later when [setting up custom domains](./set-up-custom-domains.md). +Skip the `domain_config` config for now. These settings are optionally configured later when [setting up custom domains](./set-up-custom-domains.md) and when [setting up HTTPS](./https-support.md). ### 2. Set up each network From da50ed272a594a695987db04738357796dc68569 Mon Sep 17 00:00:00 2001 From: Rocket Date: Fri, 26 Apr 2024 14:28:41 -0500 Subject: [PATCH 20/89] WIP --- docs/infra/set-up-app-config.md | 2 + docs/infra/set-up-aws-account.md | 2 + docs/infra/set-up-aws-accounts.md | 4 +- docs/infra/set-up-network.md | 4 +- docs/infra/set-up-networks.md | 2 +- infra/networks/main.tf | 13 +++---- infra/project-config/networks.tf | 65 ++++++++++++++++++------------- 7 files changed, 55 insertions(+), 37 deletions(-) diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md index d0041a80e..4cce98940 100644 --- a/docs/infra/set-up-app-config.md +++ b/docs/infra/set-up-app-config.md @@ -11,6 +11,8 @@ This setup process applies to each application in the project. ## Instructions +Follow these instructions for **each application** in your project (this can be one or more). + ### 1. Configure the app-config Modify the following values in the application's `app-config` (e.g. in `/infra/app/app-config/main.tf`): diff --git a/docs/infra/set-up-aws-account.md b/docs/infra/set-up-aws-account.md index 36dbb4e44..beee685e9 100644 --- a/docs/infra/set-up-aws-account.md +++ b/docs/infra/set-up-aws-account.md @@ -14,6 +14,8 @@ The AWS account setup process will: ## Instructions +Follow these instructions for **each AWS account** you want to configure (this can be one or more). + ### 1. Make sure you're authenticated into the AWS account you want to configure The account set up sets up whatever account you're authenticated into. To see which account that is, run diff --git a/docs/infra/set-up-aws-accounts.md b/docs/infra/set-up-aws-accounts.md index 108e6bedd..80af511ed 100644 --- a/docs/infra/set-up-aws-accounts.md +++ b/docs/infra/set-up-aws-accounts.md @@ -62,8 +62,8 @@ Decide on the strategy that is appropriate for your project. ### 3. Ensure AWS account(s) have been created -For **each** AWS account you wish to use, ensure it has been created in AWS and you are able to authenticate to it. +For **each AWS account** you wish to use, ensure it has been created in AWS and you are able to authenticate to it. ### 4. Set up AWS account -For **each** AWS account you wish to use, [set up the AWS account](./set-up-aws-account.md). \ No newline at end of file +For **each AWS account** you wish to use, [set up the AWS account](./set-up-aws-account.md). \ No newline at end of file diff --git a/docs/infra/set-up-network.md b/docs/infra/set-up-network.md index 77e768e47..3b1bdb5da 100644 --- a/docs/infra/set-up-network.md +++ b/docs/infra/set-up-network.md @@ -1,6 +1,6 @@ # Set up network -The network setup process will configure and deploy network resources needed by other modules for **one network**. In particular, it will: +The network setup process will configure and deploy network resources needed by other modules for one network. In particular, it will: 1. Create a nondefault VPC 2. Create public subnets for publicly accessible resources such as the application load balancer, private subnets for the application service, and private subnets for the database. @@ -14,6 +14,8 @@ The network setup process will configure and deploy network resources needed by ## Instructions +Follow these instructions for **each network** in your project (this can be one or more). + ### 1. Make sure you're authenticated into the AWS account you want to configure The network is set up for whatever account you're authenticated into. To see which account that is, run diff --git a/docs/infra/set-up-networks.md b/docs/infra/set-up-networks.md index e5e54ed46..ad3e96f25 100644 --- a/docs/infra/set-up-networks.md +++ b/docs/infra/set-up-networks.md @@ -25,4 +25,4 @@ Skip the `domain_config` config for now. These settings are optionally configure ### 2. Set up each network -For **each network** listed in `/infra/project-config/networks.tf`, [set up the network](./set-up-network.md). \ No newline at end of file +For **each network** in `/infra/project-config/networks.tf`, [set up the network](./set-up-network.md). \ No newline at end of file diff --git a/infra/networks/main.tf b/infra/networks/main.tf index 3e615d1ab..554f0decb 100644 --- a/infra/networks/main.tf +++ b/infra/networks/main.tf @@ -69,10 +69,9 @@ module "network" { has_external_non_aws_service = local.has_external_non_aws_service } -# Uncomment the following section to enable custom domain management. -# module "domain" { -# source = "../modules/domain" -# name = local.domain_config.hosted_zone -# manage_dns = local.domain_config.manage_dns -# certificate_configs = local.domain_config.certificate_configs -# } +module "domain" { + source = "../modules/domain" + name = local.domain_config.hosted_zone + manage_dns = local.domain_config.manage_dns + certificate_configs = local.domain_config.certificate_configs +} diff --git a/infra/project-config/networks.tf b/infra/project-config/networks.tf index 69051ead8..566d43a7b 100644 --- a/infra/project-config/networks.tf +++ b/infra/project-config/networks.tf @@ -3,37 +3,51 @@ locals { dev = { database_subnet_group_name = "dev" + + # To configure custom domains, configure the following: + # - Set `manage_dns` to `true` + # - Set the `hosted_zone` value. + # A hosted zone represents a domain and all of its subdomains. For example, a + # hosted zone of foo.domain.com includes foo.domain.com, bar.foo.domain.com, etc. domain_config = { - manage_dns = true - # Placeholder value for the hosted zone - # A hosted zone represents a domain and all of its subdomains. For example, a - # hosted zone of foo.domain.com includes foo.domain.com, bar.foo.domain.com, etc. - hosted_zone = "hosted.zone.for.dev.network.com" - - certificate_configs = { - # Example certificate configuration for a certificate that is managed by the project - # "sub.domain.com" = { - # source = "issued" - # } - - # Example certificate configuration for a certificate that is issued elsewhere and imported into the project - # (currently not supported, will be supported via https://github.com/navapbc/template-infra/issues/559) - # "platform-test-dev.navateam.com" = { - # source = "imported" - # private_key_ssm_name = "/certificates/sub.domain.com/private-key" - # certificate_body_ssm_name = "/certificates/sub.domain.com/certificate-body" - # } - } + # Set to `true` to + manage_dns = false + hosted_zone = "" + + # To configure HTTPS support, custom domains must be configured. + certificate_configs = {} } + + # domain_config = { + # manage_dns = true + # # Placeholder value for the hosted zone + # # A hosted zone represents a domain and all of its subdomains. For example, a + # # hosted zone of foo.domain.com includes foo.domain.com, bar.foo.domain.com, etc. + # hosted_zone = "hosted.zone.for.dev.network.com" + + # certificate_configs = { + # # Example certificate configuration for a certificate that is managed by the project + # # "sub.domain.com" = { + # # source = "issued" + # # } + + # # Example certificate configuration for a certificate that is issued elsewhere and imported into the project + # # (currently not supported, will be supported via https://github.com/navapbc/template-infra/issues/559) + # # "platform-test-dev.navateam.com" = { + # # source = "imported" + # # private_key_ssm_name = "/certificates/sub.domain.com/private-key" + # # certificate_body_ssm_name = "/certificates/sub.domain.com/certificate-body" + # # } + # } + # } } staging = { database_subnet_group_name = "staging" domain_config = { - manage_dns = true - hosted_zone = "hosted.zone.for.staging.network.com" - + manage_dns = false + hosted_zone = "" certificate_configs = {} } } @@ -42,9 +56,8 @@ locals { database_subnet_group_name = "prod" domain_config = { - manage_dns = true - hosted_zone = "hosted.zone.for.prod.network.com" - + manage_dns = false + hosted_zone = "" certificate_configs = {} } } From 6396ff486e9a13bdce2fb763657897d28fb65cf4 Mon Sep 17 00:00:00 2001 From: Rocket Date: Fri, 26 Apr 2024 16:24:35 -0500 Subject: [PATCH 21/89] Continue to refine multi-* language --- docs/infra/set-up-app-config.md | 6 ++---- docs/infra/set-up-aws-account.md | 4 ++-- docs/infra/set-up-aws-accounts.md | 2 +- docs/infra/set-up-network.md | 4 ++-- docs/infra/set-up-networks.md | 7 +++---- 5 files changed, 10 insertions(+), 13 deletions(-) diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md index 4cce98940..f115af84d 100644 --- a/docs/infra/set-up-app-config.md +++ b/docs/infra/set-up-app-config.md @@ -1,8 +1,8 @@ # Set up application config -The application config setup process will configure the application. These values will be used in subsequent infra setup steps to determine which resources to deploy and how they will be configured. +Follow these instructions for **each application** in your project (you can have one or more in your project). -This setup process applies to each application in the project. +The application config setup process will configure one application. These values will be used in subsequent infra setup steps to determine which resources to deploy and how they will be configured. ## Prerequisites @@ -11,8 +11,6 @@ This setup process applies to each application in the project. ## Instructions -Follow these instructions for **each application** in your project (this can be one or more). - ### 1. Configure the app-config Modify the following values in the application's `app-config` (e.g. in `/infra/app/app-config/main.tf`): diff --git a/docs/infra/set-up-aws-account.md b/docs/infra/set-up-aws-account.md index beee685e9..6bbfbd654 100644 --- a/docs/infra/set-up-aws-account.md +++ b/docs/infra/set-up-aws-account.md @@ -1,5 +1,7 @@ # Set up AWS account +Follow these instructions for **each AWS account** you want to configure (you can have one or more in your project). + The AWS account setup process will: 1. Create the [Terraform backend](https://www.terraform.io/language/settings/backends/configuration) resources needed to store Terraform's infrastructure state files. The project uses an [S3 backend](https://www.terraform.io/language/settings/backends/s3). @@ -14,8 +16,6 @@ The AWS account setup process will: ## Instructions -Follow these instructions for **each AWS account** you want to configure (this can be one or more). - ### 1. Make sure you're authenticated into the AWS account you want to configure The account set up sets up whatever account you're authenticated into. To see which account that is, run diff --git a/docs/infra/set-up-aws-accounts.md b/docs/infra/set-up-aws-accounts.md index 80af511ed..ee8466723 100644 --- a/docs/infra/set-up-aws-accounts.md +++ b/docs/infra/set-up-aws-accounts.md @@ -4,7 +4,7 @@ The infrastructure supports managing resources across different [AWS accounts](h The environment-to-AWS-account mapping is specified for each application deployed by this infrastructure. This means that multiple applications can share AWS accounts or be deployed into different AWS accounts. -The AWS account setup process will: +The AWS accounts setup process will: 1. Help you decide on your environments 2. Help you decide on your AWS account strategy diff --git a/docs/infra/set-up-network.md b/docs/infra/set-up-network.md index 3b1bdb5da..d1e2ca1b4 100644 --- a/docs/infra/set-up-network.md +++ b/docs/infra/set-up-network.md @@ -1,5 +1,7 @@ # Set up network +Follow these instructions for **each network** in your project (you can have one or more in your project). + The network setup process will configure and deploy network resources needed by other modules for one network. In particular, it will: 1. Create a nondefault VPC @@ -14,8 +16,6 @@ The network setup process will configure and deploy network resources needed by ## Instructions -Follow these instructions for **each network** in your project (this can be one or more). - ### 1. Make sure you're authenticated into the AWS account you want to configure The network is set up for whatever account you're authenticated into. To see which account that is, run diff --git a/docs/infra/set-up-networks.md b/docs/infra/set-up-networks.md index ad3e96f25..49d0e1eb0 100644 --- a/docs/infra/set-up-networks.md +++ b/docs/infra/set-up-networks.md @@ -1,10 +1,9 @@ # Set up networks -The network setup process will configure and deploy network resources needed by other modules. In particular, for each network, it will: +The networks setup process will: -1. Create a nondefault VPC -2. Create public subnets for publicly accessible resources such as the application load balancer, private subnets for the application service, and private subnets for the database. -3. Create VPC endpoints for the AWS services needed by ECS Fargate to fetch the container image and log to AWS CloudWatch. If your application has a database, it will also create VPC endpoints for the AWS services needed by the database layer and a security group to contain those VPC endpoints. +1. Configure project networks to match the project's environment mapping +3. Configure all networks ## Prerequisites From 4ea13f183634bc9efb16b4631966ab862ac01e0a Mon Sep 17 00:00:00 2001 From: Rocket Date: Fri, 26 Apr 2024 16:50:48 -0500 Subject: [PATCH 22/89] Rework build repository docs --- docs/infra/set-up-app-build-repository.md | 28 +++++++++++++++++------ docs/infra/set-up-app-config.md | 16 +++++++++++-- docs/infra/set-up-networks.md | 2 +- infra/project-config/networks.tf | 8 +++---- 4 files changed, 40 insertions(+), 14 deletions(-) diff --git a/docs/infra/set-up-app-build-repository.md b/docs/infra/set-up-app-build-repository.md index 48e633370..9d17e9efc 100644 --- a/docs/infra/set-up-app-build-repository.md +++ b/docs/infra/set-up-app-build-repository.md @@ -1,17 +1,31 @@ # Set up application build repository -The application build repository setup process will create infrastructure resources needed to store built release candidate artifacts used to deploy the application to an environment. +Follow these instructions for **each application** in your project (you can have one or more in your project). -This setup process applies to each application in the project. +The application build repository setup process will create infrastructure resources needed to store built release candidate artifacts used to deploy the application to an environment. -## Requirements +## Prerequisites Before setting up the application's build repository you'll need to have: -1. [Set up the AWS account(s)](./set-up-aws-accounts.md) -2. [Configure the app](/infra/app/app-config/main.tf) +1. You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md). +2. You'll need to have [configured the application](/infra/app/app-config/main.tf). + +### 1. Make sure you're authenticated into the AWS account where you want to deploy resources shared across environments + +The account set up sets up whatever account you're authenticated into. To see which account that is, run + +```bash +aws sts get-caller-identity +``` + +To see a more human readable account alias instead of the account, run + +```bash +aws iam list-account-aliases +``` -## 1. Configure backend +### 2. Configure backend To create the tfbackend file for the build repository using the backend configuration values from your current AWS account, run @@ -21,7 +35,7 @@ make infra-configure-app-build-repository APP_NAME= Pass in the name of the app folder within `infra`. By default this is `app`. -## 2. Create build repository resources +### 2. Create build repository resources Now run the following commands to create the resources, making sure to verify the plan before confirming the apply. diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md index f115af84d..f0ac21614 100644 --- a/docs/infra/set-up-app-config.md +++ b/docs/infra/set-up-app-config.md @@ -11,7 +11,19 @@ The application config setup process will configure one application. These value ## Instructions -### 1. Configure the app-config +### 1. Optionally, rename the application + +By default, the application module is named `app` in [`/infra/app`](/infra/app). You may want to rename the application to something project-specific. + +### 2. Optionally, create a terraform module for each additional application + +If you have multiple applications in your project, create a terraform module for each additional application by running + +```bash +curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/download-app-module.sh | bash -s +``` + +### 3. Configure the app-config Modify the following values in the application's `app-config` (e.g. in `/infra/app/app-config/main.tf`): @@ -21,7 +33,7 @@ Modify the following values in the application's `app-config` (e.g. in `/infra/a * Set `has_incident_management_service` to `true` or `false` to indicate whether the application should integrate with an incident management service. By default, this is set to `false`. * Set the `account_names_by_environment` hash to map environments to AWS accounts. See [set up AWS accounts](./set-up-aws-accounts.md) for more information. -### 2. Configure each environment +### 4. Configure each environment Within the `app-config` directory (e.g. `infra/app/app-config`), each environment configured in the `environments` array in the previous step should have its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. diff --git a/docs/infra/set-up-networks.md b/docs/infra/set-up-networks.md index 49d0e1eb0..864a5da26 100644 --- a/docs/infra/set-up-networks.md +++ b/docs/infra/set-up-networks.md @@ -2,7 +2,7 @@ The networks setup process will: -1. Configure project networks to match the project's environment mapping +1. Configure project networks, including to be compatible with the project's environment mapping 3. Configure all networks ## Prerequisites diff --git a/infra/project-config/networks.tf b/infra/project-config/networks.tf index 566d43a7b..11795b22e 100644 --- a/infra/project-config/networks.tf +++ b/infra/project-config/networks.tf @@ -46,8 +46,8 @@ locals { database_subnet_group_name = "staging" domain_config = { - manage_dns = false - hosted_zone = "" + manage_dns = false + hosted_zone = "" certificate_configs = {} } } @@ -56,8 +56,8 @@ locals { database_subnet_group_name = "prod" domain_config = { - manage_dns = false - hosted_zone = "" + manage_dns = false + hosted_zone = "" certificate_configs = {} } } From 39a8023bd23822b619017d9304a18c37db40db67 Mon Sep 17 00:00:00 2001 From: Rocket Date: Fri, 26 Apr 2024 17:41:19 -0500 Subject: [PATCH 23/89] Address app-service --- docs/feature-flags.md | 2 +- docs/infra/set-up-app-build-repository.md | 13 ++-- docs/infra/set-up-app-config.md | 10 ++- docs/infra/set-up-app-env.md | 60 ---------------- docs/infra/set-up-app-service.md | 70 +++++++++++++++++++ docs/infra/set-up-aws-account.md | 4 +- docs/infra/set-up-aws-accounts.md | 2 +- docs/infra/set-up-database.md | 54 ++++++++------ ...t-up-environment-variables-and-secrets.md} | 2 + docs/infra/set-up-network.md | 6 +- docs/infra/set-up-networks.md | 4 +- infra/README.md | 18 ++--- infra/app/app-config/feature-flags.tf | 7 +- template-only-docs/set-up-ci.md | 4 +- 14 files changed, 149 insertions(+), 107 deletions(-) delete mode 100644 docs/infra/set-up-app-env.md create mode 100644 docs/infra/set-up-app-service.md rename docs/infra/{environment-variables-and-secrets.md => set-up-environment-variables-and-secrets.md} (97%) diff --git a/docs/feature-flags.md b/docs/feature-flags.md index 63bf6a483..82a9549b8 100644 --- a/docs/feature-flags.md +++ b/docs/feature-flags.md @@ -8,7 +8,7 @@ This project leverages [Amazon CloudWatch Evidently](https://docs.aws.amazon.com ## Creating feature flags -The list of feature flags for an application is defined in the `feature_flags` property in its app-config module (in `/infra/[app_name]/app-config/feature-flags.tf`). To create a new feature flag, add a new string to that list. To remove a feature flag, remove the feature flag from the list. The set of feature flags will be updated on the next terraform apply of the service layer, or during the next deploy of the application. +The list of feature flags for an application is defined in the `feature_flags` property in its app-config module (in `/infra//app-config/feature-flags.tf`). To create a new feature flag, add a new string to that list. To remove a feature flag, remove the feature flag from the list. The set of feature flags will be updated on the next terraform apply of the service layer, or during the next deploy of the application. ## Querying feature flags in the application diff --git a/docs/infra/set-up-app-build-repository.md b/docs/infra/set-up-app-build-repository.md index 9d17e9efc..71fb56095 100644 --- a/docs/infra/set-up-app-build-repository.md +++ b/docs/infra/set-up-app-build-repository.md @@ -1,6 +1,6 @@ # Set up application build repository -Follow these instructions for **each application** in your project (you can have one or more in your project). +Follow these instructions for **each application** in your project (you can have one or more in your project). Skip this step if the application does not need a build repository (such as if the project uses pre-built images hosted in an external container repository). The application build repository setup process will create infrastructure resources needed to store built release candidate artifacts used to deploy the application to an environment. @@ -8,12 +8,15 @@ The application build repository setup process will create infrastructure resour Before setting up the application's build repository you'll need to have: -1. You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md). -2. You'll need to have [configured the application](/infra/app/app-config/main.tf). +* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) +* You'll need to have [configured the application](/infra/app/app-config/main.tf) +* You'll need to have [set up the network(s)](./set-up-networks.md) + +## Instructions ### 1. Make sure you're authenticated into the AWS account where you want to deploy resources shared across environments -The account set up sets up whatever account you're authenticated into. To see which account that is, run +This set up takes effect in whatever account you're authenticated into. To see which account that is, run ```bash aws sts get-caller-identity @@ -35,7 +38,7 @@ make infra-configure-app-build-repository APP_NAME= Pass in the name of the app folder within `infra`. By default this is `app`. -### 2. Create build repository resources +### 3. Create build repository resources Now run the following commands to create the resources, making sure to verify the plan before confirming the apply. diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md index f0ac21614..a896e1920 100644 --- a/docs/infra/set-up-app-config.md +++ b/docs/infra/set-up-app-config.md @@ -25,17 +25,19 @@ curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only ### 3. Configure the app-config -Modify the following values in the application's `app-config` (e.g. in `/infra/app/app-config/main.tf`): +Modify the following values in the application's `app-config/main.tf` (e.g. in `/infra//app-config/main.tf`): * Set the `environments` array to list the names of the environments for this application. By default, this is set to `["dev", "staging", "prod"]`. -* Set `has_database` to `true` or `false` to indicate whether or not the application has a database to integrate with. This setting determines whether or not to create VPC endpoints needed by the database layer. By default, this is set to `false`. +* Set `has_database` to `true` or `false` to indicate whether or not the application has a database to integrate with. This setting determines whether or not to create VPC endpoints needed by the database layer and whether. By default, this is set to `false`. * Set `has_external_non_aws_service` to `true` or `false` to indicate whether or not your application makes calls to an external non-AWS service. This setting determines whether or not to create NAT gateways, which allows the service in the private subnet to make requests to the internet. By default, this is set to `false`. * Set `has_incident_management_service` to `true` or `false` to indicate whether the application should integrate with an incident management service. By default, this is set to `false`. * Set the `account_names_by_environment` hash to map environments to AWS accounts. See [set up AWS accounts](./set-up-aws-accounts.md) for more information. +Optionally, to use [feature flags](/docs/feature-flags.md), modify the values in the applications `app-config/feature-flags.tf` (e.g. in `/infra//app-config/feature-flags.tf`). + ### 4. Configure each environment -Within the `app-config` directory (e.g. `infra/app/app-config`), each environment configured in the `environments` array in the previous step should have its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. +Within the `app-config` directory (e.g. `infra//app-config`), each environment configured in the `environments` array in the previous step should have its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. In each environment config file, modify the following values: @@ -43,3 +45,5 @@ In each environment config file, modify the following values: * Set `network_name`. By default, it should match the name of the environment. This mapping ensures that each network is configured appropriately based on the application(s) in that network (see `local.apps_in_network` in `/infra/networks/main.tf`). Failure to set the network name properly means that the network layer may not receive the correct application configurations for `has_database` and `has_external_non_aws_service`. * Skip `domain_name` for now * Skip `enable_https` for now + +When configuring the production environment, make sure to update the `service_cpu`, `service_memory`, and `service_desired_instance_count` settings based on the project's needs. If your application is sensitive to performance, consider doing a load test. diff --git a/docs/infra/set-up-app-env.md b/docs/infra/set-up-app-env.md deleted file mode 100644 index 1773a860f..000000000 --- a/docs/infra/set-up-app-env.md +++ /dev/null @@ -1,60 +0,0 @@ -# Set up application environment - -The application environment setup process will: - -1. Configure a new application environment and create the infrastructure resources for the application in that environment - -## Requirements - -Before setting up the application's environments you'll need to have: - -1. [A compatible application in the app folder](https://github.com/navapbc/template-infra/blob/main/template-only-docs/application-requirements.md) -2. [Configure the app](/infra/app/app-config/main.tf). Make sure you update `has_database` to `true` or `false` depending on whether or not your application has a database to integrate with. - 1. If you're configuring your production environment, make sure to update the `service_cpu`, `service_memory`, and `service_desired_instance_count` settings based on the project's needs. If your application is sensitive to performance, consider doing a load test. -3. [Create a nondefault VPC to be used by the application](./set-up-network.md) -4. (If the application has a database) [Set up the database for the application](./set-up-database.md) -5. (If you have an incident management service) [Set up monitoring](./set-up-monitoring-alerts.md) -6. [Set up the application build repository](./set-up-app-build-repository.md) - -## 1. Configure backend - -To create the tfbackend and tfvars files for the new application environment, run - -```bash -make infra-configure-app-service APP_NAME=app ENVIRONMENT= -``` - -`APP_NAME` needs to be the name of the application folder within the `infra` folder. It defaults to `app`. -`ENVIRONMENT` needs to be the name of the environment you are creating. This will create a file called `.s3.tfbackend` in the `infra/app/service` module directory. - -Depending on the value of `has_database` in the [app-config module](/infra/app/app-config/main.tf), the application will be configured with or without database access. - -## 2. Build and publish the application to the application build repository - -Before creating the application resources, you'll need to first build and publish at least one image to the application build repository. - -There are two ways to do this: - -1. Trigger the "Build and Publish" workflow from your repo's GitHub Actions tab. This option requires that the `role-to-assume` GitHub workflow variable has already been setup as part of the overall infra account setup process. -1. Alternatively, run the following from the root directory. This option can take much longer than the GitHub workflow, depending on your machine's architecture. - - ```bash - make release-build APP_NAME=app - make release-publish APP_NAME=app - ``` - -Copy the image tag name that was published. You'll need this in the next step. - -## 3. Create application resources with the image tag that was published - -Now run the following commands to create the resources, using the image tag that was published from the previous step. Review the terraform before confirming "yes" to apply the changes. - -```bash -TF_CLI_ARGS_apply="-var=image_tag=" make infra-update-app-service APP_NAME=app ENVIRONMENT= -``` - -## 4. Configure monitoring alerts - -Configure email alerts, external incident management service integration and additional Cloudwatch Alerts. -[Configure monitoring module](./set-up-monitoring-alerts.md) - diff --git a/docs/infra/set-up-app-service.md b/docs/infra/set-up-app-service.md new file mode 100644 index 000000000..eac8b8f81 --- /dev/null +++ b/docs/infra/set-up-app-service.md @@ -0,0 +1,70 @@ +# Set up application service + +Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. + +The application service setup process will: + +1. Configure an ECS Fargate Service and Task to host the application +2. Creates a load balancer for the application +2. Create an S3 bucket for general object storage +3. Set up CloudWatch for logging, monitoring, and alerts +3. Optionally, configure CloudWatch Evidently to support [feature flags](/docs/feature-flags.md) + +## Prerequisites + +* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) +* You'll need to have [configured the application](/infra/app/app-config/main.tf) +* You'll need to have [set up the network(s)](./set-up-networks.md) +* Optionally, if you need a container build repository, you'll need to have [set up the build repository](./set-up-app-build-repository.md) +* Optionally, if you need a database for the application, you'll need to have [set up the database](./set-up-database.md) + +## Instructions + +### 1. Make sure you're authenticated into the AWS account where you want to deploy this environment + +This set up takes effect in whatever account you're authenticated into. To see which account that is, run + +```bash +aws sts get-caller-identity +``` + +To see a more human readable account alias instead of the account, run + +```bash +aws iam list-account-aliases +``` + +### 2. Configure backend + +To create the tfbackend and tfvars files for the new application environment, run + +```bash +make infra-configure-app-service APP_NAME= ENVIRONMENT= +``` + +`APP_NAME` needs to be the name of the application folder within the `infra` folder. It defaults to `app`. +`ENVIRONMENT` needs to be the name of the environment you are creating. This will create a file called `.s3.tfbackend` in the `infra//service` module directory. + +### 3. Build and publish the application to the application build repository + +Before creating the application resources, you'll need to first build and publish at least one image to the application build repository. + +There are two ways to do this: + +1. Trigger the "Build and Publish" workflow from your repo's GitHub Actions tab. This option requires that the `role-to-assume` GitHub workflow variable has already been setup as part of the overall infra account setup process. +1. Alternatively, run the following from the root directory. This option can take much longer than the GitHub workflow, depending on your machine's architecture. + + ```bash + make release-build APP_NAME= + make release-publish APP_NAME= + ``` + +Copy the image tag name that was published. You'll need this in the next step. + +### 4. Create application resources with the image tag that was published + +Now run the following commands to create the resources, using the image tag that was published from the previous step. Review the terraform before confirming "yes" to apply the changes. + +```bash +TF_CLI_ARGS_apply="-var=image_tag=" make infra-update-app-service APP_NAME= ENVIRONMENT= +``` \ No newline at end of file diff --git a/docs/infra/set-up-aws-account.md b/docs/infra/set-up-aws-account.md index 6bbfbd654..f8eca00e6 100644 --- a/docs/infra/set-up-aws-account.md +++ b/docs/infra/set-up-aws-account.md @@ -10,8 +10,8 @@ The AWS account setup process will: ## Prerequisites -* You'll need to have [set up infrastructure tools](./set-up-infrastructure-tools.md), like Terraform, AWS CLI, and AWS authentication. -* You'll also need to make sure the [project is configured](/infra/project-config/main.tf). +* You'll need to have [set up infrastructure tools](./set-up-infrastructure-tools.md), like Terraform, AWS CLI, and AWS authentication +* You'll also need to make sure the [project is configured](/infra/project-config/main.tf) * You'll need to have [decided on your environments and AWS account strategy](./set-up-aws-accounts.md) ## Instructions diff --git a/docs/infra/set-up-aws-accounts.md b/docs/infra/set-up-aws-accounts.md index ee8466723..30be6acce 100644 --- a/docs/infra/set-up-aws-accounts.md +++ b/docs/infra/set-up-aws-accounts.md @@ -12,7 +12,7 @@ The AWS accounts setup process will: ## Prerequisites -* You'll need to have [set up infrastructure tools](./set-up-infrastructure-tools.md), like Terraform, AWS CLI, and AWS authentication. +* You'll need to have [set up infrastructure tools](./set-up-infrastructure-tools.md), like Terraform, AWS CLI, and AWS authentication ## Instructions diff --git a/docs/infra/set-up-database.md b/docs/infra/set-up-database.md index d5cb6c66e..6dde844cc 100644 --- a/docs/infra/set-up-database.md +++ b/docs/infra/set-up-database.md @@ -1,21 +1,39 @@ # Set up database +Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. Skip this step if the application does not need a database. + The database setup process will: 1. Configure and deploy an application database cluster using [Amazon Aurora Serverless V2](https://aws.amazon.com/rds/aurora/serverless/) -2. Create a [PostgreSQL schema](https://www.postgresql.org/docs/current/ddl-schemas.html) `app` to contain tables used by the application. +2. Create a [PostgreSQL schema](https://www.postgresql.org/docs/current/ddl-schemas.html) `app` to contain tables used by the application 3. Create an IAM policy that allows IAM roles with that policy attached to [connect to the database using IAM authentication](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.html) -4. Create an [AWS Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html), the "role manager", for provisioning the [PostgreSQL database users](https://www.postgresql.org/docs/8.0/user-manag.html) that will be used by the application service and by the migrations task. -5. Invoke the role manager function to create the `app` and `migrator` Postgres users. +4. Create an [AWS Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html), the "role manager", for provisioning the [PostgreSQL database users](https://www.postgresql.org/docs/8.0/user-manag.html) that will be used by the application service and by the migrations task +5. Invoke the role manager function to create the `app` and `migrator` Postgres users + +## Prerequisites + +* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) +* You'll need to have [configured the application](/infra/app/app-config/main.tf) +* You'll need to have [set up the network(s)](./set-up-networks.md) +* You'll need to have [pip](https://pypi.org/project/pip/) installed (pip is needed to download dependencies for the role manager Lambda function) + +## Instructions + +### 1. Make sure you're authenticated into the AWS account where you want to deploy this environment + +This set up takes effect in whatever account you're authenticated into. To see which account that is, run -## Requirements +```bash +aws sts get-caller-identity +``` -Before setting up the database you'll need to have: +To see a more human readable account alias instead of the account, run -1. [Set up the AWS account](./set-up-aws-account.md) -2. pip installed (pip is needed to download dependencies for the role manager Lambda function) +```bash +aws iam list-account-aliases +``` -## 1. Configure backend +### 2. Configure backend To create the tfbackend file for the new application environment, run @@ -24,22 +42,22 @@ make infra-configure-app-database APP_NAME= ENVIRONMENT= ``` `APP_NAME` needs to be the name of the application folder within the `infra` folder. By default, this is `app`. -`ENVIRONMENT` needs to be the name of the environment you are creating. This will create a file called `.s3.tfbackend` in the `infra/app/service` module directory. +`ENVIRONMENT` needs to be the name of the environment you are creating. This will create a file called `.s3.tfbackend` in the `infra//service` module directory. -## 2. Create database resources +### 3. Create database resources Now run the following commands to create the resources. Review the terraform before confirming "yes" to apply the changes. This can take over 5 minutes. ```bash -make infra-update-app-database APP_NAME=app ENVIRONMENT= +make infra-update-app-database APP_NAME= ENVIRONMENT= ``` -## 3. Create Postgres users +### 4. Create Postgres users Trigger the role manager Lambda function that was created in the previous step in order to create the application and migrator Postgres users. ```bash -make infra-update-app-database-roles APP_NAME=app ENVIRONMENT= +make infra-update-app-database-roles APP_NAME= ENVIRONMENT= ``` The Lambda function's response should describe the resulting PostgreSQL roles and groups that are configured in the database. It should look like a minified version of the following: @@ -65,7 +83,7 @@ The Lambda function's response should describe the resulting PostgreSQL roles an } ``` -### Important note on Postgres table permissions +#### Important note on Postgres table permissions Before creating migrations that create tables, first create a migration that includes the following SQL command (or equivalent if your migrations are written in a general purpose programming language): @@ -77,12 +95,8 @@ This will cause all future tables created by the `migrator` user to automaticall Why is this needed? The reason is because the `migrator` role will be used by the migration task to run database migrations (creating tables, altering tables, etc.), while the `app` role will be used by the web service to access the database. Moreover, in Postgres, new tables won't automatically be accessible by roles other than the creator unless specifically granted, even if those other roles have usage access to the schema that the tables are created in. In other words if the `migrator` user created a new table `foo` in the `app` schema, the `app` user will not have automatically be able to access it by default. -## 4. Check that database roles have been configured properly +### 5. Check that database roles have been configured properly ```bash -make infra-check-app-database-roles APP_NAME=app ENVIRONMENT= +make infra-check-app-database-roles APP_NAME= ENVIRONMENT= ``` - -## Set up application environments - -Once you set up the deployment process, you can proceed to [set up the application service](./set-up-app-env.md) diff --git a/docs/infra/environment-variables-and-secrets.md b/docs/infra/set-up-environment-variables-and-secrets.md similarity index 97% rename from docs/infra/environment-variables-and-secrets.md rename to docs/infra/set-up-environment-variables-and-secrets.md index f05d39df6..d6bb61d69 100644 --- a/docs/infra/environment-variables-and-secrets.md +++ b/docs/infra/set-up-environment-variables-and-secrets.md @@ -1,5 +1,7 @@ # Environment variables and secrets +Follow these instructions for **each application** in your project (you can have one or more in your project). + Applications follow [12-factor app](https://12factor.net/) principles to [store configuration as environment variables](https://12factor.net/config). The infrastructure provides some of these environment variables automatically, such as environment variables to authenticate as the ECS task role, environment variables for database access, and environment variables for accessing document storage. However, many applications require extra custom environment variables for application configuration and for access to secrets. This document describes how to configure application-specific environment variables and secrets. It also describes how to override those environment variables for a specific environment. ## Application-specific extra environment variables diff --git a/docs/infra/set-up-network.md b/docs/infra/set-up-network.md index d1e2ca1b4..8919c368c 100644 --- a/docs/infra/set-up-network.md +++ b/docs/infra/set-up-network.md @@ -10,9 +10,9 @@ The network setup process will configure and deploy network resources needed by ## Prerequisites -* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md). -* You'll need to have [configured all applications](./set-up-app-config.md). -* You'll need to have [configured the project's networks](./set-up-networks.md). +* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) +* You'll need to have [configured all applications](./set-up-app-config.md) +* You'll need to have [configured the project's networks](./set-up-networks.md) ## Instructions diff --git a/docs/infra/set-up-networks.md b/docs/infra/set-up-networks.md index 864a5da26..eaffffc91 100644 --- a/docs/infra/set-up-networks.md +++ b/docs/infra/set-up-networks.md @@ -7,8 +7,8 @@ The networks setup process will: ## Prerequisites -* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md). -* You'll need to have [configured all applications](./set-up-app-config.md). +* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) +* You'll need to have [configured all applications](./set-up-app-config.md) ## Instructions diff --git a/infra/README.md b/infra/README.md index a2d03c778..69a9c007c 100644 --- a/infra/README.md +++ b/infra/README.md @@ -45,7 +45,7 @@ This project has the following AWS environments: - `staging` - `prod` -The environments share the same root modules but will have different configurations. Backend configuration is saved as [`.tfbackend`](https://developer.hashicorp.com/terraform/language/settings/backends/configuration#file) files. Most `.tfbackend` files are named after the environment. For example, the `[APP_NAME]/service` infrastructure resources for the `dev` environment are configured via `dev.s3.tfbackend`. Resources for a module that are shared across environments, such as the build-repository, use `shared.s3.tfbackend`. Resources that are shared across the entire account (e.g. /infra/accounts) use `..s3.tfbackend`. +The environments share the same root modules but will have different configurations. Backend configuration is saved as [`.tfbackend`](https://developer.hashicorp.com/terraform/language/settings/backends/configuration#file) files. Most `.tfbackend` files are named after the environment. For example, the `/service` infrastructure resources for the `dev` environment are configured via `dev.s3.tfbackend`. Resources for a module that are shared across environments, such as the build-repository, use `shared.s3.tfbackend`. Resources that are shared across the entire account (e.g. /infra/accounts) use `..s3.tfbackend`. ### 🔀 Project workflow @@ -72,13 +72,15 @@ To set up this project for the first time, if it has never been deployed to the 1. [Configure the application](/docs/infra/set-up-app-config.md) 5. [Set up the network(s)](/docs/infra/set-up-networks.md) 6. For each application: - 1. [Set up application build repository](/docs/infra/set-up-app-build-repository.md) - 2. [Set up application database](/docs/infra/set-up-database.md) - 3. [Set up application environment](/docs/infra/set-up-app-env.md) - 4. [Configure environment variables and secrets](/docs/infra/environment-variables-and-secrets.md) - 5. [Set up background jobs](/docs/infra/background-jobs.md) - 6. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md). You can also come back to setting up custom domains at a later time. - 7. Optionally, [configure HTTPS support](/docs/infra/https-support.md). You can also come back to setting up HTTPS support at a later time. + 1. Optionally, [configure environment variables and secrets](/docs/infra/set-up-environment-variables-and-secrets.md) + 2. Optionally, [set up application build repository](/docs/infra/set-up-app-build-repository.md) + 3. For each environment: + 1. Optionally, [set up application database](/docs/infra/set-up-database.md) + 2. [Set up application service](/docs/infra/set-up-app-service.md) + 3. (If you have an incident management service) [Set up monitoring](./set-up-monitoring-alerts.md) + 4. [Set up background jobs](/docs/infra/background-jobs.md) + 4. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md). You can also come back to setting up custom domains at a later time. + 5. Optionally, [configure HTTPS support](/docs/infra/https-support.md). You can also come back to setting up HTTPS support at a later time. ### 🆕 New developer diff --git a/infra/app/app-config/feature-flags.tf b/infra/app/app-config/feature-flags.tf index 6224b2538..a92d9f9c1 100644 --- a/infra/app/app-config/feature-flags.tf +++ b/infra/app/app-config/feature-flags.tf @@ -1,3 +1,8 @@ locals { - feature_flags = ["foo", "bar"] + # Add feature flags as strings to the array. + # For instance: + # feature_flags = ["foo", "bar"] + # See the /docs/feature-flags.md for more information. + feature_flags = [] + } diff --git a/template-only-docs/set-up-ci.md b/template-only-docs/set-up-ci.md index 7264adfad..11fa5e56b 100644 --- a/template-only-docs/set-up-ci.md +++ b/template-only-docs/set-up-ci.md @@ -1,5 +1,7 @@ # Set up CI +--- @TODO + ## Static analysis checks CI should automatically be set up once the CI files in `.github/workflows` are committed and pushed to the remote repository in GitHub. @@ -8,7 +10,7 @@ Some checks are disabled until you've completed certain setup steps: ### After setting up the application environment -After [setting up the app environment](/docs/infra/set-up-app-env.md): +After [setting up the app environment](/docs/infra/set-up-app-service.md): - Uncomment the infra end-to-end tests by searching for `!!` in [ci-infra-service.yml](/.github/workflows/ci-infra-service.yml). You can verify that CI is running and passing by clicking into the Actions tab in GitHub. Note that this repo only contains CI for infra. Application CI (`ci-app.yml`) is included as part of the application templates. - Uncomment the push trigger in [cd-app.yml](/.github/workflows/cd-app.yml) From 27b3c87c1957c19d068c2ec598c67d763c83a174 Mon Sep 17 00:00:00 2001 From: Rocket Date: Fri, 26 Apr 2024 18:04:04 -0500 Subject: [PATCH 24/89] Work on monitoring --- docs/infra/set-up-app-build-repository.md | 10 ++- ...-up-database.md => set-up-app-database.md} | 5 ++ docs/infra/set-up-app-monitoring-alerts.md | 66 +++++++++++++++++++ docs/infra/set-up-app-service.md | 3 +- docs/infra/set-up-monitoring-alerts.md | 29 -------- infra/README.md | 10 +-- 6 files changed, 85 insertions(+), 38 deletions(-) rename docs/infra/{set-up-database.md => set-up-app-database.md} (95%) create mode 100644 docs/infra/set-up-app-monitoring-alerts.md delete mode 100644 docs/infra/set-up-monitoring-alerts.md diff --git a/docs/infra/set-up-app-build-repository.md b/docs/infra/set-up-app-build-repository.md index 71fb56095..c62848124 100644 --- a/docs/infra/set-up-app-build-repository.md +++ b/docs/infra/set-up-app-build-repository.md @@ -1,6 +1,6 @@ # Set up application build repository -Follow these instructions for **each application** in your project (you can have one or more in your project). Skip this step if the application does not need a build repository (such as if the project uses pre-built images hosted in an external container repository). +Follow these instructions for **each application** in your project (you can have one or more in your project). The application build repository setup process will create infrastructure resources needed to store built release candidate artifacts used to deploy the application to an environment. @@ -36,7 +36,7 @@ To create the tfbackend file for the build repository using the backend configur make infra-configure-app-build-repository APP_NAME= ``` -Pass in the name of the app folder within `infra`. By default this is `app`. +`APP_NAME` needs to be the name of the application folder within the `infra` folder. It defaults to `app`. ### 3. Create build repository resources @@ -44,4 +44,8 @@ Now run the following commands to create the resources, making sure to verify th ```bash make infra-update-app-build-repository APP_NAME= -``` \ No newline at end of file +``` + +## If the application does not need a build repository + +If the application does not need a build repository (such as if the project uses pre-built images hosted in an external container repository), delete the application's build repository module (e.g. `/infra//build-repository`). \ No newline at end of file diff --git a/docs/infra/set-up-database.md b/docs/infra/set-up-app-database.md similarity index 95% rename from docs/infra/set-up-database.md rename to docs/infra/set-up-app-database.md index 6dde844cc..69e51aa21 100644 --- a/docs/infra/set-up-database.md +++ b/docs/infra/set-up-app-database.md @@ -42,6 +42,7 @@ make infra-configure-app-database APP_NAME= ENVIRONMENT= ``` `APP_NAME` needs to be the name of the application folder within the `infra` folder. By default, this is `app`. + `ENVIRONMENT` needs to be the name of the environment you are creating. This will create a file called `.s3.tfbackend` in the `infra//service` module directory. ### 3. Create database resources @@ -100,3 +101,7 @@ Why is this needed? The reason is because the `migrator` role will be used by th ```bash make infra-check-app-database-roles APP_NAME= ENVIRONMENT= ``` + +## If the application does not need a database + +If the application does not need a database (such as if the project uses an alternative for data persistence), delete the application's database module (e.g. `/infra//database`). \ No newline at end of file diff --git a/docs/infra/set-up-app-monitoring-alerts.md b/docs/infra/set-up-app-monitoring-alerts.md new file mode 100644 index 000000000..e6852d554 --- /dev/null +++ b/docs/infra/set-up-app-monitoring-alerts.md @@ -0,0 +1,66 @@ +# Set up monitoring notifications + +Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. Skip this step if the application does not need a monitoring alerts. + +The monitoring module defines metric-based alerting policies that provides awareness into issues with the cloud application. The module supports integration with external incident management tools like Splunk-On-Call or Pagerduty. It also supports email alerts. + +## Prerequisites + +* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) +* You'll need to have [configured the application](/infra/app/app-config/main.tf) +* You'll need to have [set up the network(s)](./set-up-networks.md) +* Optionally, if you need a container build repository, you'll need to have [set up the build repository](./set-up-app-build-repository.md) + +## Instructions + +### To set up email alerts + +When any of the alerts described by the module are triggered, a notification will be send to all emails specified in the `email_alerts_subscription_list`. + +#### 1. Update the application's service layer + +In the application's service module (e.g. `/infra//service/main.tf`), uncomment the `email_alerts_subscription_list` key and add the emails that should be notified. + +For example: + +``` +module "monitoring" { + source = "../../modules/monitoring" + email_alerts_subscription_list = ["email1@email.com", "email2@email.com"] + ... +} +``` + +#### 2. Update the application service + +To apply the changes, run the following command for each environment: + +```bash +make infra-update-app-service APP_NAME= ENVIRONMENT=` +``` + +`APP_NAME` needs to be the name of the application folder within the `infra` folder. By default, this is `app`. + +`ENVIRONMENT` needs to be the name of the environment to update. + +### To set up external incident management service integration + +#### 1. Update the application config + +Modify the `has_incident_management_service` in the application's `app-config/main.tf` (e.g. in `/infra//app-config/main.tf`) and set it to `true`. + +#### 2. Add the external url as a secret + +Get the integration URL for the incident management service and store it in AWS SSM Parameter Store by running the following command for each environment: + +```bash +make infra-configure-monitoring-secrets APP_NAME= ENVIRONMENT= URL= +``` + +#### 3. Update the application service + +To apply the changes, run the following command for each environment: + +```bash +make infra-update-app-service APP_NAME= ENVIRONMENT=` +``` diff --git a/docs/infra/set-up-app-service.md b/docs/infra/set-up-app-service.md index eac8b8f81..c0eef3e71 100644 --- a/docs/infra/set-up-app-service.md +++ b/docs/infra/set-up-app-service.md @@ -16,7 +16,7 @@ The application service setup process will: * You'll need to have [configured the application](/infra/app/app-config/main.tf) * You'll need to have [set up the network(s)](./set-up-networks.md) * Optionally, if you need a container build repository, you'll need to have [set up the build repository](./set-up-app-build-repository.md) -* Optionally, if you need a database for the application, you'll need to have [set up the database](./set-up-database.md) +* Optionally, if you need a database for the application, you'll need to have [set up the database](./set-up-app-database.md) ## Instructions @@ -43,6 +43,7 @@ make infra-configure-app-service APP_NAME= ENVIRONMENT= ``` `APP_NAME` needs to be the name of the application folder within the `infra` folder. It defaults to `app`. + `ENVIRONMENT` needs to be the name of the environment you are creating. This will create a file called `.s3.tfbackend` in the `infra//service` module directory. ### 3. Build and publish the application to the application build repository diff --git a/docs/infra/set-up-monitoring-alerts.md b/docs/infra/set-up-monitoring-alerts.md deleted file mode 100644 index 0619d529e..000000000 --- a/docs/infra/set-up-monitoring-alerts.md +++ /dev/null @@ -1,29 +0,0 @@ -# Set up monitoring notifications - -## Overview - -The monitoring module defines metric-based alerting policies that provides awareness into issues with the cloud application. The module supports integration with external incident management tools like Splunk-On-Call or Pagerduty. It also supports email alerts. - -### Set up email alerts. - -1. Add the `email_alerts_subscription_list` variable to the monitoring module call in the service layer - -For example: -``` -module "monitoring" { - source = "../../modules/monitoring" - email_alerts_subscription_list = ["email1@email.com", "email2@email.com"] - ... -} -``` -2. Run `make infra-update-app-service APP_NAME= ENVIRONMENT=` to apply the changes to each environment. -When any of the alerts described by the module are triggered notification will be send to all email specified in the `email_alerts_subscription_list` - -### Set up External incident management service integration. - -1. Set setting `has_incident_management_service = true` in app-config/main.tf -2. Get the integration URL for the incident management service and store it in AWS SSM Parameter Store by running the following command for each environment: -``` -make infra-configure-monitoring-secrets APP_NAME= ENVIRONMENT= URL= -``` -3. Run `make infra-update-app-service APP_NAME= ENVIRONMENT=` to apply the changes to each environment. diff --git a/infra/README.md b/infra/README.md index 69a9c007c..20f335032 100644 --- a/infra/README.md +++ b/infra/README.md @@ -73,12 +73,12 @@ To set up this project for the first time, if it has never been deployed to the 5. [Set up the network(s)](/docs/infra/set-up-networks.md) 6. For each application: 1. Optionally, [configure environment variables and secrets](/docs/infra/set-up-environment-variables-and-secrets.md) - 2. Optionally, [set up application build repository](/docs/infra/set-up-app-build-repository.md) + 2. [Set up application build repository](/docs/infra/set-up-app-build-repository.md) 3. For each environment: - 1. Optionally, [set up application database](/docs/infra/set-up-database.md) - 2. [Set up application service](/docs/infra/set-up-app-service.md) - 3. (If you have an incident management service) [Set up monitoring](./set-up-monitoring-alerts.md) - 4. [Set up background jobs](/docs/infra/background-jobs.md) + 1. [Set up application database](/docs/infra/set-up-app-database.md) + 2. [Set up application service](/docs/infra/set-up-app-service.md) + 3. Optionally, [set up monitoring alerts](/docs/infra/set-up-app-monitoring-alerts.md) + 4. Optionally, [set up background jobs](/docs/infra/background-jobs.md) 4. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md). You can also come back to setting up custom domains at a later time. 5. Optionally, [configure HTTPS support](/docs/infra/https-support.md). You can also come back to setting up HTTPS support at a later time. From 162799cd77ec80703aed948a78ec3b427fdfcaf4 Mon Sep 17 00:00:00 2001 From: Rocket Date: Fri, 26 Apr 2024 18:21:12 -0500 Subject: [PATCH 25/89] Work on custom domains --- docs/infra/https-support.md | 4 +- docs/infra/set-up-app-database.md | 2 +- docs/infra/set-up-app-monitoring-alerts.md | 4 +- docs/infra/set-up-custom-domains.md | 43 +++++++++++----------- infra/README.md | 4 +- template-only-docs/set-up-ci.md | 2 +- 6 files changed, 31 insertions(+), 28 deletions(-) diff --git a/docs/infra/https-support.md b/docs/infra/https-support.md index aee72d411..c15f6b98f 100644 --- a/docs/infra/https-support.md +++ b/docs/infra/https-support.md @@ -1,4 +1,6 @@ -# HTTPS support +# Set up HTTPS + +--- @TODO docs refactor Production systems will want to use HTTPS rather than HTTP to prevent man-in-the-middle attacks. This document describes how HTTPS is configured. This process will: diff --git a/docs/infra/set-up-app-database.md b/docs/infra/set-up-app-database.md index 69e51aa21..316b14095 100644 --- a/docs/infra/set-up-app-database.md +++ b/docs/infra/set-up-app-database.md @@ -1,6 +1,6 @@ # Set up database -Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. Skip this step if the application does not need a database. +Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. The database setup process will: diff --git a/docs/infra/set-up-app-monitoring-alerts.md b/docs/infra/set-up-app-monitoring-alerts.md index e6852d554..64fcc77d9 100644 --- a/docs/infra/set-up-app-monitoring-alerts.md +++ b/docs/infra/set-up-app-monitoring-alerts.md @@ -36,7 +36,7 @@ module "monitoring" { To apply the changes, run the following command for each environment: ```bash -make infra-update-app-service APP_NAME= ENVIRONMENT=` +make infra-update-app-service APP_NAME= ENVIRONMENT= ``` `APP_NAME` needs to be the name of the application folder within the `infra` folder. By default, this is `app`. @@ -62,5 +62,5 @@ make infra-configure-monitoring-secrets APP_NAME= ENVIRONMENT= ENVIRONMENT=` +make infra-update-app-service APP_NAME= ENVIRONMENT= ``` diff --git a/docs/infra/set-up-custom-domains.md b/docs/infra/set-up-custom-domains.md index cfaac37e2..307306253 100644 --- a/docs/infra/set-up-custom-domains.md +++ b/docs/infra/set-up-custom-domains.md @@ -1,36 +1,33 @@ # Custom domains +Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. Skip this step if the application does not need custom domains. + Production systems will want to set up custom domains to route internet traffic to their application services rather than using AWS-generated hostnames for the load balancers or the CDN. This document describes how to configure custom domains. -**For each network**, the custom domain setup process will: +The custom domain setup process will: 1. Create an [Amazon Route 53 hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) to manage DNS records for a domain and subdomains 2. Create a DNS A (address) records to route traffic from a custom domain to an application's load balancer ## Prerequisites -* You'll need to have registered custom domain(s) with a domain registrar (e.g. Namecheap, GoDaddy, Google Domains, etc.). -* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md). -* You'll need to have [configured all applications](./set-up-app-config.md). -* You'll need to have [set up the networks](./set-up-network.md) that you want to add the custom domain to. +* You'll need to have registered custom domain(s) with a domain registrar (e.g. Namecheap, GoDaddy, Google Domains, etc.) +* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) +* You'll need to have [configured all applications](./set-up-app-config.md) +* You'll need to have [set up the networks](./set-up-network.md) that you want to add the custom domain to +* You'll need to have [set up the application service](./set-up-app-service.md) ## Instructions -Follow these instructions for **each network** you want to setup custom domains for. - ### 1. Set hosted zone in domain configuration The custom domain configuration is defined as a `domain_config` object in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf). A [hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) represents a domain and all of its subdomains. For example, a hosted zone of `platform-test.navateam.com` includes `platform-test.navateam.com`, `cdn.platform-test.navateam.com`, `notifications.platform-test.navateam.com`, `foo.bar.platform-test.navateam.com`, etc. -**For each network** you want to use a custom domain, modify the `hosted_zone` value in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) to match the custom domain (or a subdomain of the custom domain) that you registered. Each `hosted_zone` value must be different. - -### 2. Enable custom domains +**For each network** you want to use a custom domain, set the `hosted_zone` value in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) to match the custom domain (or a subdomain of the custom domain) that you registered. -In the [network module](/infra/networks/main.tf), enable custom domains by uncommenting the `module "domain"` section. - -### 3. Update the network layer to create the hosted zones +### 2. Update the network layer to create the hosted zones **For each network** you that you added a custom domain to in Step 1, run the following command to create the hosted zone specified in the domain configuration: @@ -69,24 +66,28 @@ Run the following command to verify that DNS requests are being served by the ho nslookup -type=NS ``` -### 4. Configure custom domain for your application +### 4. Create DNS A (address) records to route traffic from the custom domain to the application's load balancer + +Within the `app-config` directory (e.g. `infra//app-config`), each environment has its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. -Define the `domain_name` for each of the application environments in the `app-config` module. The `domain_name` must be either the same as the `hosted_zone` or a subdomain of the `hosted_zone`. For example, if your hosted zone is `platform-test.navateam.com`, then `platform-test.navateam.com` and `cdn.platform-test.navateam.com` are both valid values for `domain_name`. +In each environment config file, define the `domain_name`. -### 5. Create DNS A (address) records to route traffic from the custom domain to your application's load balancer +The `domain_name` must be either the same as the `hosted_zone` or a subdomain of the `hosted_zone`. For example, if your hosted zone is `platform-test.navateam.com`, then `platform-test.navateam.com` and `cdn.platform-test.navateam.com` are both valid values for `domain_name`. -If created after.... +### 5. Update the application service -Run the following command to create the A record that routes traffic from the custom domain to the application's load balancer. +To apply the changes, run the following command for each environment: ```bash make infra-update-app-service APP_NAME= ENVIRONMENT= ``` -### 6. Repeat for each application +`APP_NAME` needs to be the name of the application folder within the `infra` folder. By default, this is `app`. -If you have multiple applications in the same network, repeat steps 4 and 5 for each application. +`ENVIRONMENT` needs to be the name of the environment. ## Externally managed DNS -If you plan to manage DNS records outside of the project, then set `network_configs[*].domain_config.manage_dns = false` in [the networks section of the project-config module](/infra/project-config/networks.tf). +--- @TODO create ticket to make this a local that is derived from hosted_zone + +If DNS records are managed externally outside of the project, set `network_configs[*].domain_config.manage_dns = false` in [the networks section of the project-config module](/infra/project-config/networks.tf). diff --git a/infra/README.md b/infra/README.md index 20f335032..e3fbf8745 100644 --- a/infra/README.md +++ b/infra/README.md @@ -77,8 +77,8 @@ To set up this project for the first time, if it has never been deployed to the 3. For each environment: 1. [Set up application database](/docs/infra/set-up-app-database.md) 2. [Set up application service](/docs/infra/set-up-app-service.md) - 3. Optionally, [set up monitoring alerts](/docs/infra/set-up-app-monitoring-alerts.md) - 4. Optionally, [set up background jobs](/docs/infra/background-jobs.md) + 3. Optionally, [set up application monitoring alerts](/docs/infra/set-up-app-monitoring-alerts.md) + 4. Optionally, [set up application background jobs](/docs/infra/background-jobs.md) 4. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md). You can also come back to setting up custom domains at a later time. 5. Optionally, [configure HTTPS support](/docs/infra/https-support.md). You can also come back to setting up HTTPS support at a later time. diff --git a/template-only-docs/set-up-ci.md b/template-only-docs/set-up-ci.md index 11fa5e56b..18078861d 100644 --- a/template-only-docs/set-up-ci.md +++ b/template-only-docs/set-up-ci.md @@ -1,6 +1,6 @@ # Set up CI ---- @TODO +--- @TODO needs doc refactoring ## Static analysis checks From e4e196b61d4e3b14274bd3d46eafd94fcdd01e0a Mon Sep 17 00:00:00 2001 From: Rocket Date: Fri, 26 Apr 2024 18:25:54 -0500 Subject: [PATCH 26/89] Adjust indentation --- infra/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/infra/README.md b/infra/README.md index e3fbf8745..ea07999da 100644 --- a/infra/README.md +++ b/infra/README.md @@ -79,8 +79,8 @@ To set up this project for the first time, if it has never been deployed to the 2. [Set up application service](/docs/infra/set-up-app-service.md) 3. Optionally, [set up application monitoring alerts](/docs/infra/set-up-app-monitoring-alerts.md) 4. Optionally, [set up application background jobs](/docs/infra/background-jobs.md) - 4. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md). You can also come back to setting up custom domains at a later time. - 5. Optionally, [configure HTTPS support](/docs/infra/https-support.md). You can also come back to setting up HTTPS support at a later time. + 5. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md) + 6. Optionally, [configure HTTPS support](/docs/infra/https-support.md) ### 🆕 New developer @@ -88,7 +88,7 @@ To get set up as a new developer to the project, if it has already been deployed 1. [Set up your infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md) 2. [Review how to make changes to infrastructure](/docs/infra/making-infra-changes.md) -3. (Optional) Set up a [terraform workspace](/docs/infra/intro-to-terraform-workspaces.md) +3. Optionally, set up a [terraform workspace](/docs/infra/intro-to-terraform-workspaces.md) ## 📇 Additional reading From 3d1ca3143f6a31563e8200fa75dd23acb3fbc93c Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 30 Apr 2024 11:24:28 -0700 Subject: [PATCH 27/89] Wordsmith --- docs/infra/set-up-app-config.md | 2 ++ docs/infra/set-up-infrastructure-tools.md | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md index a896e1920..2c196ab9e 100644 --- a/docs/infra/set-up-app-config.md +++ b/docs/infra/set-up-app-config.md @@ -19,6 +19,8 @@ By default, the application module is named `app` in [`/infra/app`](/infra/app). If you have multiple applications in your project, create a terraform module for each additional application by running +--- @TODO create + ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/download-app-module.sh | bash -s ``` diff --git a/docs/infra/set-up-infrastructure-tools.md b/docs/infra/set-up-infrastructure-tools.md index e27477f36..c7e12fca5 100644 --- a/docs/infra/set-up-infrastructure-tools.md +++ b/docs/infra/set-up-infrastructure-tools.md @@ -1,6 +1,6 @@ # Set up infrastructure developer tools -If you are contributing to infrastructure, you will need to complete these setup steps. +If you are working on the infrastructure, you will need to complete these setup steps. ## Prerequisites @@ -8,7 +8,7 @@ If you are contributing to infrastructure, you will need to complete these setup [Terraform](https://www.terraform.io/) is an infrastructure as code (IaC) tool that allows you to build, change, and version infrastructure safely and efficiently. This includes both low-level components like compute instances, storage, and networking, as well as high-level components like DNS entries and SaaS features. -You may need different versions of Terraform since different projects may require different versions of Terraform. The best way to manage Terraform versions is with [Terraform Version Manager](https://github.com/tfutils/tfenv). +You may need different versions of Terraform since different projects may require different versions of Terraform. The best way to manage Terraform versions is with [Terraform Version Manager (tfenv)](https://github.com/tfutils/tfenv). To install via [Homebrew](https://brew.sh/) From eee2a2454c891f252641ada7f45cc8eb1081f1f1 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 30 Apr 2024 11:38:59 -0700 Subject: [PATCH 28/89] Add more clarity --- docs/infra/set-up-app-config.md | 22 ++++++---------------- docs/infra/set-up-aws-accounts.md | 2 ++ 2 files changed, 8 insertions(+), 16 deletions(-) diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md index 2c196ab9e..851b9fca2 100644 --- a/docs/infra/set-up-app-config.md +++ b/docs/infra/set-up-app-config.md @@ -11,21 +11,11 @@ The application config setup process will configure one application. These value ## Instructions -### 1. Optionally, rename the application +### 1. (Optional) Rename the application -By default, the application module is named `app` in [`/infra/app`](/infra/app). You may want to rename the application to something project-specific. +By default, the application module is named `app` in [`/infra`](/infra/). You may want to rename the application to something project-specific. -### 2. Optionally, create a terraform module for each additional application - -If you have multiple applications in your project, create a terraform module for each additional application by running - ---- @TODO create - -```bash -curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/download-app-module.sh | bash -s -``` - -### 3. Configure the app-config +### 2. Configure the app-config Modify the following values in the application's `app-config/main.tf` (e.g. in `/infra//app-config/main.tf`): @@ -35,11 +25,11 @@ Modify the following values in the application's `app-config/main.tf` (e.g. in ` * Set `has_incident_management_service` to `true` or `false` to indicate whether the application should integrate with an incident management service. By default, this is set to `false`. * Set the `account_names_by_environment` hash to map environments to AWS accounts. See [set up AWS accounts](./set-up-aws-accounts.md) for more information. -Optionally, to use [feature flags](/docs/feature-flags.md), modify the values in the applications `app-config/feature-flags.tf` (e.g. in `/infra//app-config/feature-flags.tf`). +Optionally, to use [feature flags](/docs/feature-flags.md), modify the values in the application's `app-config/feature-flags.tf` (e.g. in `/infra//app-config/feature-flags.tf`). -### 4. Configure each environment +### 3. Configure each environment -Within the `app-config` directory (e.g. `infra//app-config`), each environment configured in the `environments` array in the previous step should have its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. +Within the application's `app-config` directory (e.g. `infra//app-config`), each environment configured in the `environments` array in the previous step should have its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. In each environment config file, modify the following values: diff --git a/docs/infra/set-up-aws-accounts.md b/docs/infra/set-up-aws-accounts.md index 30be6acce..1394ec0fc 100644 --- a/docs/infra/set-up-aws-accounts.md +++ b/docs/infra/set-up-aws-accounts.md @@ -28,6 +28,8 @@ You can always add new environments or delete existing environments later. ### 2. Decide on AWS account strategy +Some resources, such as a build repository, are shared across environments. These resources can be deployed to the same AWS account as any of the other environments or they can be deployed to a separate environment. In the following diagrams, these are represented by the box labeled `shared`. + A simple project might have only one AWS account and all environments should be deployed to this environment. This mapping might look like this: ```mermaid From b28fea58a33206f645f90136dff3a6348de7608c Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 30 Apr 2024 11:44:46 -0700 Subject: [PATCH 29/89] Consistency --- docs/infra/set-up-networks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/infra/set-up-networks.md b/docs/infra/set-up-networks.md index eaffffc91..5000cd462 100644 --- a/docs/infra/set-up-networks.md +++ b/docs/infra/set-up-networks.md @@ -1,6 +1,6 @@ # Set up networks -The networks setup process will: +The networks set up process will: 1. Configure project networks, including to be compatible with the project's environment mapping 3. Configure all networks From a397335dd69de937db0c25e628dd927db8df2546 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 30 Apr 2024 11:44:52 -0700 Subject: [PATCH 30/89] Consistency --- docs/infra/set-up-app-build-repository.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/infra/set-up-app-build-repository.md b/docs/infra/set-up-app-build-repository.md index c62848124..a36172376 100644 --- a/docs/infra/set-up-app-build-repository.md +++ b/docs/infra/set-up-app-build-repository.md @@ -1,13 +1,11 @@ # Set up application build repository -Follow these instructions for **each application** in your project (you can have one or more in your project). +Follow these instructions for **each application** in your project (you can have one or more in your project). If the application does not need a build repository, skip to the bottom of this document. The application build repository setup process will create infrastructure resources needed to store built release candidate artifacts used to deploy the application to an environment. ## Prerequisites -Before setting up the application's build repository you'll need to have: - * You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) * You'll need to have [configured the application](/infra/app/app-config/main.tf) * You'll need to have [set up the network(s)](./set-up-networks.md) @@ -36,7 +34,7 @@ To create the tfbackend file for the build repository using the backend configur make infra-configure-app-build-repository APP_NAME= ``` -`APP_NAME` needs to be the name of the application folder within the `infra` folder. It defaults to `app`. +`APP_NAME` needs to be the name of the application folder within the `infra` folder. ### 3. Create build repository resources From ab98d6614945cd6e2fa9a8a23cdfe57829f7dbd4 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 30 Apr 2024 11:47:19 -0700 Subject: [PATCH 31/89] Consistency. Clarity --- docs/infra/set-up-app-database.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/infra/set-up-app-database.md b/docs/infra/set-up-app-database.md index 316b14095..8f7b0df39 100644 --- a/docs/infra/set-up-app-database.md +++ b/docs/infra/set-up-app-database.md @@ -1,6 +1,6 @@ # Set up database -Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. +Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. If the application does not need a database, skip to the bottom of this document. The database setup process will: @@ -41,7 +41,7 @@ To create the tfbackend file for the new application environment, run make infra-configure-app-database APP_NAME= ENVIRONMENT= ``` -`APP_NAME` needs to be the name of the application folder within the `infra` folder. By default, this is `app`. +`APP_NAME` needs to be the name of the application folder within the `infra` folder. `ENVIRONMENT` needs to be the name of the environment you are creating. This will create a file called `.s3.tfbackend` in the `infra//service` module directory. @@ -104,4 +104,4 @@ make infra-check-app-database-roles APP_NAME= ENVIRONMENT=/database`). \ No newline at end of file +If the application does not need a database (such as if the project uses an alternative for data persistence), delete the application's database module (e.g. `/infra//database`) and ensure that the application's `app-config` sets `has_database` to `false` (see [set up app config](./set-up-app-config.md)). \ No newline at end of file From 16047427c8b640044c0490ee8ec4603fdffb3087 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 30 Apr 2024 11:49:53 -0700 Subject: [PATCH 32/89] Fix typo --- docs/infra/set-up-app-database.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/infra/set-up-app-database.md b/docs/infra/set-up-app-database.md index 8f7b0df39..f01a92b89 100644 --- a/docs/infra/set-up-app-database.md +++ b/docs/infra/set-up-app-database.md @@ -43,7 +43,7 @@ make infra-configure-app-database APP_NAME= ENVIRONMENT= `APP_NAME` needs to be the name of the application folder within the `infra` folder. -`ENVIRONMENT` needs to be the name of the environment you are creating. This will create a file called `.s3.tfbackend` in the `infra//service` module directory. +`ENVIRONMENT` needs to be the name of the environment you are creating. This will create a file called `.s3.tfbackend` in the `infra//database` module directory. ### 3. Create database resources From c460330dceba1e93c350d6de2f14a9d6a698da3c Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 30 Apr 2024 11:56:17 -0700 Subject: [PATCH 33/89] Clarify --- docs/infra/set-up-app-service.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/infra/set-up-app-service.md b/docs/infra/set-up-app-service.md index c0eef3e71..6085d47f5 100644 --- a/docs/infra/set-up-app-service.md +++ b/docs/infra/set-up-app-service.md @@ -36,19 +36,19 @@ aws iam list-account-aliases ### 2. Configure backend -To create the tfbackend and tfvars files for the new application environment, run +To create the tfbackend and tfvars files for the new application service, run ```bash make infra-configure-app-service APP_NAME= ENVIRONMENT= ``` -`APP_NAME` needs to be the name of the application folder within the `infra` folder. It defaults to `app`. +`APP_NAME` needs to be the name of the application folder within the `infra` folder. `ENVIRONMENT` needs to be the name of the environment you are creating. This will create a file called `.s3.tfbackend` in the `infra//service` module directory. ### 3. Build and publish the application to the application build repository -Before creating the application resources, you'll need to first build and publish at least one image to the application build repository. +Before creating the application resources, you'll need to first build and publish at least one image to the application build repository. This step does not need to be run per-environment. There are two ways to do this: From 082ffd9c76d96b785e0961e700bc22eb25325d9b Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 30 Apr 2024 12:00:45 -0700 Subject: [PATCH 34/89] Clarity --- docs/infra/set-up-app-monitoring-alerts.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/docs/infra/set-up-app-monitoring-alerts.md b/docs/infra/set-up-app-monitoring-alerts.md index 64fcc77d9..264e14bef 100644 --- a/docs/infra/set-up-app-monitoring-alerts.md +++ b/docs/infra/set-up-app-monitoring-alerts.md @@ -1,6 +1,6 @@ # Set up monitoring notifications -Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. Skip this step if the application does not need a monitoring alerts. +Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. If the application does not need a monitoring notifications, skip to the bottom of this document. The monitoring module defines metric-based alerting policies that provides awareness into issues with the cloud application. The module supports integration with external incident management tools like Splunk-On-Call or Pagerduty. It also supports email alerts. @@ -9,7 +9,7 @@ The monitoring module defines metric-based alerting policies that provides aware * You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) * You'll need to have [configured the application](/infra/app/app-config/main.tf) * You'll need to have [set up the network(s)](./set-up-networks.md) -* Optionally, if you need a container build repository, you'll need to have [set up the build repository](./set-up-app-build-repository.md) +* If the application needs custom-built container images, you'll need to have [set up the build repository](./set-up-app-build-repository.md) ## Instructions @@ -19,6 +19,8 @@ When any of the alerts described by the module are triggered, a notification wil #### 1. Update the application's service layer +--- @TODO move to app-config + In the application's service module (e.g. `/infra//service/main.tf`), uncomment the `email_alerts_subscription_list` key and add the emails that should be notified. For example: @@ -39,7 +41,7 @@ To apply the changes, run the following command for each environment: make infra-update-app-service APP_NAME= ENVIRONMENT= ``` -`APP_NAME` needs to be the name of the application folder within the `infra` folder. By default, this is `app`. +`APP_NAME` needs to be the name of the application folder within the `infra` folder. `ENVIRONMENT` needs to be the name of the environment to update. @@ -51,7 +53,7 @@ Modify the `has_incident_management_service` in the application's `app-config/ma #### 2. Add the external url as a secret -Get the integration URL for the incident management service and store it in AWS SSM Parameter Store by running the following command for each environment: +Get the integration URL for the incident management service and store it in AWS SSM Parameter Store by running the following command ```bash make infra-configure-monitoring-secrets APP_NAME= ENVIRONMENT= URL= @@ -59,8 +61,15 @@ make infra-configure-monitoring-secrets APP_NAME= ENVIRONMENT= ENVIRONMENT= ``` + +## If the application does not need monitoring notifications + +If the application does not need a monitoring, ensure that: + +* the application's service module (e.g. `/infra//service/main.tf`) has the `email_alerts_subscription_list` commented out +* the application's `app-config` (e.g. in `/infra//app-config/main.tf`) has `has_incident_management_service` set to `false` \ No newline at end of file From 96af1c58b283cc86008a363c0f37da0d8c6d2fd8 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 30 Apr 2024 12:16:28 -0700 Subject: [PATCH 35/89] Fix typo --- docs/infra/set-up-app-monitoring-alerts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/infra/set-up-app-monitoring-alerts.md b/docs/infra/set-up-app-monitoring-alerts.md index 264e14bef..6a9ad5de6 100644 --- a/docs/infra/set-up-app-monitoring-alerts.md +++ b/docs/infra/set-up-app-monitoring-alerts.md @@ -69,7 +69,7 @@ make infra-update-app-service APP_NAME= ENVIRONMENT= ## If the application does not need monitoring notifications -If the application does not need a monitoring, ensure that: +If the application does not need monitoring notifications, ensure that: * the application's service module (e.g. `/infra//service/main.tf`) has the `email_alerts_subscription_list` commented out * the application's `app-config` (e.g. in `/infra//app-config/main.tf`) has `has_incident_management_service` set to `false` \ No newline at end of file From e6316ac27c32c667b1e8e5e8d636e831522f6ca5 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 30 Apr 2024 12:16:32 -0700 Subject: [PATCH 36/89] Clarity --- docs/infra/set-up-custom-domains.md | 52 +++++++++++++++++++++++------ 1 file changed, 42 insertions(+), 10 deletions(-) diff --git a/docs/infra/set-up-custom-domains.md b/docs/infra/set-up-custom-domains.md index 307306253..f216dc4a5 100644 --- a/docs/infra/set-up-custom-domains.md +++ b/docs/infra/set-up-custom-domains.md @@ -1,13 +1,13 @@ # Custom domains -Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. Skip this step if the application does not need custom domains. +Follow these instructions for **each network** (you can have one or more in your project) in your project. If the network or an application does not need custom domains, skip to the bottom of this document. Production systems will want to set up custom domains to route internet traffic to their application services rather than using AWS-generated hostnames for the load balancers or the CDN. This document describes how to configure custom domains. The custom domain setup process will: 1. Create an [Amazon Route 53 hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) to manage DNS records for a domain and subdomains -2. Create a DNS A (address) records to route traffic from a custom domain to an application's load balancer +2. Create DNS A (address) records to route traffic from a custom domain to an application's load balancer ## Prerequisites @@ -19,7 +19,21 @@ The custom domain setup process will: ## Instructions -### 1. Set hosted zone in domain configuration +### 1. Make sure you're authenticated into the AWS account you want to configure + +This set up takes effect in whatever account you're authenticated into. To see which account that is, run + +```bash +aws sts get-caller-identity +``` + +To see a more human readable account alias instead of the account, run + +```bash +aws iam list-account-aliases +``` + +### 2. Set hosted zone in domain configuration The custom domain configuration is defined as a `domain_config` object in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf). A [hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) represents a domain and all of its subdomains. @@ -27,15 +41,15 @@ For example, a hosted zone of `platform-test.navateam.com` includes `platform-te **For each network** you want to use a custom domain, set the `hosted_zone` value in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) to match the custom domain (or a subdomain of the custom domain) that you registered. -### 2. Update the network layer to create the hosted zones +### 3. Update the network layer to create the hosted zones -**For each network** you that you added a custom domain to in Step 1, run the following command to create the hosted zone specified in the domain configuration: +**For each network** you that you added a custom domain to in the previous step, run the following command to create the hosted zone specified in the domain configuration: ```bash make infra-update-network NETWORK_NAME= ``` -### 3. Delegate DNS requests to the newly created hosted zone +### 4. Delegate DNS requests to the newly created hosted zone You most likely registered your domain outside of this project. Using whichever service you used to register the domain name (e.g. Namecheap, GoDaddy, Google Domains, etc.), add a DNS NS (nameserver) record. Set the "name" equal to the `hosted_zone` and set the value equal to the list of hosted zone name servers that was created in the previous step. You can see the list of servers by running @@ -66,7 +80,9 @@ Run the following command to verify that DNS requests are being served by the ho nslookup -type=NS ``` -### 4. Create DNS A (address) records to route traffic from the custom domain to the application's load balancer +### 5. Create DNS A (address) records to route traffic from the custom domain to the application's load balancer + +**For each application** in the network that should use the custom domain, perform the following. Within the `app-config` directory (e.g. `infra//app-config`), each environment has its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. @@ -74,15 +90,15 @@ In each environment config file, define the `domain_name`. The `domain_name` must be either the same as the `hosted_zone` or a subdomain of the `hosted_zone`. For example, if your hosted zone is `platform-test.navateam.com`, then `platform-test.navateam.com` and `cdn.platform-test.navateam.com` are both valid values for `domain_name`. -### 5. Update the application service +### 6. Update the application service -To apply the changes, run the following command for each environment: +**For each application and each environment** in the network that should use the custom domain, apply the changes with the following command ```bash make infra-update-app-service APP_NAME= ENVIRONMENT= ``` -`APP_NAME` needs to be the name of the application folder within the `infra` folder. By default, this is `app`. +`APP_NAME` needs to be the name of the application folder within the `infra` folder. `ENVIRONMENT` needs to be the name of the environment. @@ -91,3 +107,19 @@ make infra-update-app-service APP_NAME= ENVIRONMENT= --- @TODO create ticket to make this a local that is derived from hosted_zone If DNS records are managed externally outside of the project, set `network_configs[*].domain_config.manage_dns = false` in [the networks section of the project-config module](/infra/project-config/networks.tf). + +## If a network does not need custom domains + +For each network that does not need custom domains, ensure the network's `domain_config` setting in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf). looks like this: + +```json +domain_config = { + manage_dns = false + hosted_zone = "" + certificate_configs = {} +} +``` + +## If an application does not need custom domains + +For each application that does not need custom domains, ensure that the application's `app-config/.tf` file (e.g. in `/infra//app-config/.tf`) has `domain_name` set to `""` (empty string). From f92c48af6788d8c01e4bd8c310d1e2c5d9eeb8bf Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 30 Apr 2024 12:17:59 -0700 Subject: [PATCH 37/89] Wrong syntax --- docs/infra/set-up-custom-domains.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/docs/infra/set-up-custom-domains.md b/docs/infra/set-up-custom-domains.md index f216dc4a5..607fccb60 100644 --- a/docs/infra/set-up-custom-domains.md +++ b/docs/infra/set-up-custom-domains.md @@ -102,17 +102,11 @@ make infra-update-app-service APP_NAME= ENVIRONMENT= `ENVIRONMENT` needs to be the name of the environment. -## Externally managed DNS - ---- @TODO create ticket to make this a local that is derived from hosted_zone - -If DNS records are managed externally outside of the project, set `network_configs[*].domain_config.manage_dns = false` in [the networks section of the project-config module](/infra/project-config/networks.tf). - ## If a network does not need custom domains For each network that does not need custom domains, ensure the network's `domain_config` setting in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf). looks like this: -```json +```hcl domain_config = { manage_dns = false hosted_zone = "" From 36248f4d1e0fad65a47ee4134fcebb34d9dc4f67 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 30 Apr 2024 12:23:54 -0700 Subject: [PATCH 38/89] Rename for consistency --- ...up-custom-domains.md => set-up-network-custom-domains.md} | 0 docs/infra/{https-support.md => set-up-network-https.md} | 5 ++--- docs/infra/set-up-networks.md | 2 +- infra/README.md | 4 ++-- 4 files changed, 5 insertions(+), 6 deletions(-) rename docs/infra/{set-up-custom-domains.md => set-up-network-custom-domains.md} (100%) rename docs/infra/{https-support.md => set-up-network-https.md} (86%) diff --git a/docs/infra/set-up-custom-domains.md b/docs/infra/set-up-network-custom-domains.md similarity index 100% rename from docs/infra/set-up-custom-domains.md rename to docs/infra/set-up-network-custom-domains.md diff --git a/docs/infra/https-support.md b/docs/infra/set-up-network-https.md similarity index 86% rename from docs/infra/https-support.md rename to docs/infra/set-up-network-https.md index c15f6b98f..417919b6d 100644 --- a/docs/infra/https-support.md +++ b/docs/infra/set-up-network-https.md @@ -1,6 +1,5 @@ # Set up HTTPS ---- @TODO docs refactor Production systems will want to use HTTPS rather than HTTP to prevent man-in-the-middle attacks. This document describes how HTTPS is configured. This process will: @@ -9,7 +8,7 @@ Production systems will want to use HTTPS rather than HTTP to prevent man-in-the ## Requirements -In order to set up HTTPS support you'll also need to have [set up custom domains](/docs/infra/set-up-custom-domains.md). This is because SSL/TLS certificates must be properly configured for the specific domain to support establishing secure connections. +In order to set up HTTPS support you'll also need to have [set up custom domains](/docs/infra/set-up-network-custom-domains.md). This is because SSL/TLS certificates must be properly configured for the specific domain to support establishing secure connections. ## 1. Set desired certificates in domain configuration @@ -31,7 +30,7 @@ aws acm describe-certificate --certificate-arn --query Certifi ## 4. Update `enable_https = true` in `app-config` -Update `enable_https = true` in your application's `app-config` module. You should have already set `domain_name` as part of [setting up custom domain names](/docs/infra/set-up-custom-domains.md). +Update `enable_https = true` in your application's `app-config` module. You should have already set `domain_name` as part of [setting up custom domain names](/docs/infra/set-up-network-custom-domains.md). ## 5. Attach certificate to load balancer diff --git a/docs/infra/set-up-networks.md b/docs/infra/set-up-networks.md index 5000cd462..59fa8dc04 100644 --- a/docs/infra/set-up-networks.md +++ b/docs/infra/set-up-networks.md @@ -20,7 +20,7 @@ By default there are three networks defined, one for each application environmen If you have multiple applications and want your applications in separate networks within the same AWS account, you may want to give the networks differentiating names (e.g. "foo-dev", "foo-prod", "bar-dev", "bar-prod", instead of just "dev", "prod"). -Skip the `domain_config` config for now. These settings are optionally configured later when [setting up custom domains](./set-up-custom-domains.md) and when [setting up HTTPS](./https-support.md). +Skip the `domain_config` config for now. These settings are optionally configured later when [setting up custom domains](./set-up-network-custom-domains.md) and when [setting up HTTPS](./set-up-network-https.md). ### 2. Set up each network diff --git a/infra/README.md b/infra/README.md index ea07999da..0dbaf97f1 100644 --- a/infra/README.md +++ b/infra/README.md @@ -79,8 +79,8 @@ To set up this project for the first time, if it has never been deployed to the 2. [Set up application service](/docs/infra/set-up-app-service.md) 3. Optionally, [set up application monitoring alerts](/docs/infra/set-up-app-monitoring-alerts.md) 4. Optionally, [set up application background jobs](/docs/infra/background-jobs.md) - 5. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md) - 6. Optionally, [configure HTTPS support](/docs/infra/https-support.md) +7. Optionally, [set up custom domains](/docs/infra/set-up-network-custom-domains.md) +8. Optionally, [set up HTTPS support](/docs/infra/set-up-network-https.md) ### 🆕 New developer From c4fd3a87247164d48407473922a774d6a2ed928e Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 30 Apr 2024 13:10:46 -0700 Subject: [PATCH 39/89] Fix typos --- docs/infra/set-up-app-database.md | 2 +- docs/infra/set-up-app-service.md | 2 +- docs/infra/set-up-network-custom-domains.md | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/infra/set-up-app-database.md b/docs/infra/set-up-app-database.md index f01a92b89..4dd42fee1 100644 --- a/docs/infra/set-up-app-database.md +++ b/docs/infra/set-up-app-database.md @@ -43,7 +43,7 @@ make infra-configure-app-database APP_NAME= ENVIRONMENT= `APP_NAME` needs to be the name of the application folder within the `infra` folder. -`ENVIRONMENT` needs to be the name of the environment you are creating. This will create a file called `.s3.tfbackend` in the `infra//database` module directory. +`ENVIRONMENT` needs to be the name of the environment to update. This will create a file called `.s3.tfbackend` in the `infra//database` module directory. ### 3. Create database resources diff --git a/docs/infra/set-up-app-service.md b/docs/infra/set-up-app-service.md index 6085d47f5..eb2458f38 100644 --- a/docs/infra/set-up-app-service.md +++ b/docs/infra/set-up-app-service.md @@ -44,7 +44,7 @@ make infra-configure-app-service APP_NAME= ENVIRONMENT= `APP_NAME` needs to be the name of the application folder within the `infra` folder. -`ENVIRONMENT` needs to be the name of the environment you are creating. This will create a file called `.s3.tfbackend` in the `infra//service` module directory. +`ENVIRONMENT` needs to be the name of the environment to update. This will create a file called `.s3.tfbackend` in the `infra//service` module directory. ### 3. Build and publish the application to the application build repository diff --git a/docs/infra/set-up-network-custom-domains.md b/docs/infra/set-up-network-custom-domains.md index 607fccb60..aa6851bbb 100644 --- a/docs/infra/set-up-network-custom-domains.md +++ b/docs/infra/set-up-network-custom-domains.md @@ -100,11 +100,11 @@ make infra-update-app-service APP_NAME= ENVIRONMENT= `APP_NAME` needs to be the name of the application folder within the `infra` folder. -`ENVIRONMENT` needs to be the name of the environment. +`ENVIRONMENT` needs to be the name of the environment to update. ## If a network does not need custom domains -For each network that does not need custom domains, ensure the network's `domain_config` setting in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf). looks like this: +For each network that does not need custom domains, ensure the network's `domain_config` setting in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) looks like this: ```hcl domain_config = { From f3061c863fbcb7f1c4804a243848cc7d3b7effc8 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 30 Apr 2024 13:10:52 -0700 Subject: [PATCH 40/89] Update network docs --- docs/infra/set-up-network-custom-domains.md | 2 +- docs/infra/set-up-network-https.md | 63 +++++++++++++++++---- 2 files changed, 54 insertions(+), 11 deletions(-) diff --git a/docs/infra/set-up-network-custom-domains.md b/docs/infra/set-up-network-custom-domains.md index aa6851bbb..20fb938fb 100644 --- a/docs/infra/set-up-network-custom-domains.md +++ b/docs/infra/set-up-network-custom-domains.md @@ -1,4 +1,4 @@ -# Custom domains +# Set up custom domains Follow these instructions for **each network** (you can have one or more in your project) in your project. If the network or an application does not need custom domains, skip to the bottom of this document. diff --git a/docs/infra/set-up-network-https.md b/docs/infra/set-up-network-https.md index 417919b6d..adfadbcc5 100644 --- a/docs/infra/set-up-network-https.md +++ b/docs/infra/set-up-network-https.md @@ -1,22 +1,43 @@ # Set up HTTPS +Follow these instructions for **each network** (you can have one or more in your project) in your project. If the network or an application does not need HTTPS, skip to the bottom of this document. Production systems will want to use HTTPS rather than HTTP to prevent man-in-the-middle attacks. This document describes how HTTPS is configured. This process will: 1. Issue an SSL/TLS certificate using Amazon Certificate Manager (ACM) for each domain that we want to support HTTPS 2. Associate the certificate with the application's load balancer so that the load balancer can serve HTTPS requests intended for that domain -## Requirements +## Prerequisites -In order to set up HTTPS support you'll also need to have [set up custom domains](/docs/infra/set-up-network-custom-domains.md). This is because SSL/TLS certificates must be properly configured for the specific domain to support establishing secure connections. +* You'll need to have [set up custom domains](./set-up-network-custom-domains.md) and met all of those prerequisites -## 1. Set desired certificates in domain configuration +This is because SSL/TLS certificates must be properly configured for the specific domain to support establishing secure connections. -For each custom domain you want to set up in the network, define a certificate configuration object and set the `source` to `issued`. You'll probably want at least one custom domain for each application/service in the network. The custom domain must be either the same as the hosted zone or a subdomain of the hosted zone. +## Instructions -## 2. Update the network layer to issue the certificates +### 1. Make sure you're authenticated into the AWS account you want to configure -Run the following command to issue SSL/TLS certificates for each custom domain you configured +This set up takes effect in whatever account you're authenticated into. To see which account that is, run + +```bash +aws sts get-caller-identity +``` + +To see a more human readable account alias instead of the account, run + +```bash +aws iam list-account-aliases +``` + +### 2. Set desired certificates in domain configuration + +**For each network** you want to configure, modify the network in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) to set the `certificate_configs` key. + +Set the `source` of the domain or subdomain to `issued`. + +### 3. Update the network layer to issue the certificates + +**For each network** you configured in the previous step, run the following command to issue SSL/TLS certificates ```bash make infra-update-network NETWORK_NAME= @@ -28,14 +49,36 @@ Run the following command to check the status of a certificate (replace ` --query Certificate.Status ``` -## 4. Update `enable_https = true` in `app-config` +### 4. Update `enable_https = true` in `app-config` -Update `enable_https = true` in your application's `app-config` module. You should have already set `domain_name` as part of [setting up custom domain names](/docs/infra/set-up-network-custom-domains.md). +**For each application and environment** that should use HTTPS, perform the following. -## 5. Attach certificate to load balancer +Within the `app-config` directory (e.g. `infra//app-config`), each environment has its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. -Run the following command to attach the SSL/TLS certificate to the load balancer +In each environment config file, set `enable_https` to `true`. This will attach the SSL/TLS certificate to the load balancer. + +You should have already set `domain_name` as part of [setting up custom domain names](/docs/infra/set-up-network-custom-domains.md). + +### 5. Attach certificate to load balancer + +**For each application and environment** that should use HTTPS, apply the changes in the previous step by running ```bash make infra-update-app-service APP_NAME= ENVIRONMENT= ``` + +`APP_NAME` needs to be the name of the application folder within the `infra` folder. + +`ENVIRONMENT` needs to be the name of the environment to update. + +## If a network does not need HTTPS + +**⚠️ This is not advised** for any network containing a production environment. + +For each network that does not need custom domains, ensure the network's `certificate_configs` setting in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) is `{}` (empty hash). + +## If an application does not need HTTPS + +**⚠️ This is not advised** for an application deployed to a production environment. + +For each application that does not need HTTPS, ensure that the application's `app-config/.tf` file (e.g. in `/infra//app-config/.tf`) has `enable_https` set to `false`. \ No newline at end of file From aea69849bbeaa6a0b4d7c0deda5d23a524104a2a Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 30 Apr 2024 15:34:03 -0700 Subject: [PATCH 41/89] Update CI/CD --- README.md | 2 +- docs/infra/set-up-app-monitoring-alerts.md | 2 -- template-only-docs/set-up-cd.md | 4 +++- template-only-docs/set-up-ci.md | 12 +++++------- 4 files changed, 9 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index c5cabbaa1..3cfdd8ac1 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ Now you're ready to set up the various pieces of your infrastructure. After downloading and installing the template into your project: 1. Follow the steps in [infra/README.md](/infra/README.md) to setup the infrastructure for your application. -2. After setting up AWS resources, you can [set up GitHub Actions workflows](./template-only-docs/set-up-ci.md). +2. After setting up AWS resources, you can [set up GitHub Actions workflows for continuous integration](./template-only-docs/set-up-ci.md). 3. After configuring GitHub Actions, you can [set up continuous deployment](./template-only-docs/set-up-cd.md). 4. At any point, [set up your team workflow](./template-only-docs/set-up-team-workflow.md). diff --git a/docs/infra/set-up-app-monitoring-alerts.md b/docs/infra/set-up-app-monitoring-alerts.md index 6a9ad5de6..ee324acb5 100644 --- a/docs/infra/set-up-app-monitoring-alerts.md +++ b/docs/infra/set-up-app-monitoring-alerts.md @@ -19,8 +19,6 @@ When any of the alerts described by the module are triggered, a notification wil #### 1. Update the application's service layer ---- @TODO move to app-config - In the application's service module (e.g. `/infra//service/main.tf`), uncomment the `email_alerts_subscription_list` key and add the emails that should be notified. For example: diff --git a/template-only-docs/set-up-cd.md b/template-only-docs/set-up-cd.md index d94437717..d9d6785b7 100644 --- a/template-only-docs/set-up-cd.md +++ b/template-only-docs/set-up-cd.md @@ -1,6 +1,8 @@ # Set up CD -Once you have set up your application environments, you can enable continuous deployment in [cd-app.yml](../.github/workflows/cd-app.yml) by searching for `!!` and following the instructions to: +Once you have set up your application(s), you can enable continuous deployment. Each application should have a CD Github Actions workflow (e.g. `/.github/workflows/cd-`). + +In each `/.github/workflows/cd-`, search for `!!` and following the instructions to: 1. Update the `role-to-assume` with the GitHub actions ARN. 2. Uncomment the `on: push: ["main]` workflow trigger. This will trigger the deployment workflow on every merge to `main`. diff --git a/template-only-docs/set-up-ci.md b/template-only-docs/set-up-ci.md index 18078861d..09b2f6504 100644 --- a/template-only-docs/set-up-ci.md +++ b/template-only-docs/set-up-ci.md @@ -1,17 +1,15 @@ # Set up CI ---- @TODO needs doc refactoring - ## Static analysis checks CI should automatically be set up once the CI files in `.github/workflows` are committed and pushed to the remote repository in GitHub. Some checks are disabled until you've completed certain setup steps: -### After setting up the application environment +### After completing infra setup -After [setting up the app environment](/docs/infra/set-up-app-service.md): +After completing the [infra setup](/infra/README.md#instructions): -- Uncomment the infra end-to-end tests by searching for `!!` in [ci-infra-service.yml](/.github/workflows/ci-infra-service.yml). You can verify that CI is running and passing by clicking into the Actions tab in GitHub. Note that this repo only contains CI for infra. Application CI (`ci-app.yml`) is included as part of the application templates. -- Uncomment the push trigger in [cd-app.yml](/.github/workflows/cd-app.yml) -- If you setup your AWS account in a different region than `us-east-1`, update the `aws-region` workflow settings to match your region. +* Uncomment the infra end-to-end tests by searching for `!!` in [ci-infra-service.yml](/.github/workflows/ci-infra-service.yml). You can verify that CI is running and passing by clicking into the Actions tab in GitHub. + * Note that this repo only contains CI for infra. If you're using one of the [Platform application templates](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), then the application CI (`/.github/workflows/ci-app.yml`) is already included. Otherwise, you'll need to create one. +* If you setup your AWS account in a different region than `us-east-1`, update the `aws-region` workflow settings in [`/.github/workflows/check-infra-auth.yml`](/.github/workflows/check-infra-auth.yml) to match your region. From 4b58d92ad077813555ebc02b2ff7212ad39b20f8 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 2 May 2024 12:29:29 -0700 Subject: [PATCH 42/89] Add docs and script for installing multiple apps --- template-only-bin/download-and-install-app.sh | 48 +++++++++++++++++++ template-only-docs/multiple-applications.md | 38 +++++++++++++++ 2 files changed, 86 insertions(+) create mode 100755 template-only-bin/download-and-install-app.sh create mode 100644 template-only-docs/multiple-applications.md diff --git a/template-only-bin/download-and-install-app.sh b/template-only-bin/download-and-install-app.sh new file mode 100755 index 000000000..f2341b413 --- /dev/null +++ b/template-only-bin/download-and-install-app.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# ----------------------------------------------------------------------------- +# This script renames the application template in your project. +# Run this script in your project's root directory. +# +# Positional parameters: +# NEW_NAME (required) - the new name for the application, in either snake- or kebab-case +# OLD_NAME (optional) – the old name for the application, in either snake- or kebab-case +# Defaults to the template's short name (e.g. app-rails) +# ----------------------------------------------------------------------------- +set -euo pipefail + +APP_NAME=$1 +CURRENT_VERSION=$(cat .template-version) + +# Use kebab-case +APP_NAME_KEBAB=$(echo $APP_NAME | tr "_" "-") + +# Helper to get the correct sed -i behavior for both GNU sed and BSD sed (installed by default on macOS) +# Hat tip: https://stackoverflow.com/a/38595160 +sedi () { + sed --version >/dev/null 2>&1 && sed -i -- "$@" || sed -i "" "$@" +} +# Export the function so it can be used later on +export -f sedi + +echo "Cloning template-infra..." +git clone https://github.com/navapbc/template-infra.git + +echo "Switching to this project's current version of the template..." +cd template-infra +git checkout "$CURRENT_VERSION" +cd - >& /dev/null + +echo "Creating a terraform module for a new application..." +cp -r template-infra/infra/app infra/"$APP_NAME_KEBAB" + +echo "Setting up new application CI/CD..." +cp template-infra/.github/workflows/cd-app.yml .github/workflows/"cd-$APP_NAME_KEBAB.yml" +LC_ALL=C sedi "s/app\([\s\"\/\-]\)/$APP_NAME_KEBAB\1/g" .github/workflows/"cd-$APP_NAME_KEBAB.yml" + +cp template-infra/.github/workflows/ci-app-vulnerability-scans.yml .github/workflows/"ci-$APP_NAME_KEBAB-vulnerability-scans.yml" +LC_ALL=C sedi "s/app\([\s\"\/\-]\)/$APP_NAME_KEBAB\1/g" .github/workflows/"ci-$APP_NAME_KEBAB-vulnerability-scans.yml" + +echo "Cleaning up template-infra folder..." +rm -fr template-infra + +echo "...Done." \ No newline at end of file diff --git a/template-only-docs/multiple-applications.md b/template-only-docs/multiple-applications.md new file mode 100644 index 000000000..95ed10d3a --- /dev/null +++ b/template-only-docs/multiple-applications.md @@ -0,0 +1,38 @@ +# Multiple Applications + +This infrastructure supports multiple deployment and CI/CD for projects with multiple applications. By default, the infrastructure assumes the project only has one application, named `app`. However, it's straightforward to include additional applications. + +## Prerequisites + +* None + +## Instructions + +### 1. Ensure the application meets the Application Requirements + +In order to use this infrastructure, the application must meets the [application requirements](/template-only-docs/application-requirements.md). + +### 2. Add the application to the root directory + +Add the application's source code to a folder that lives in the project root folder, such as `/second-app`. + +⚠️ Warning: In general, it's best to use a short, descriptive one-word name because some AWS resources have character limits. If you must use multiple words, use hyphens (not underscores) to separate each word. + +### 3. Add infrastructure support for the application + +Run the following to install the application + +```bash +curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/download-and-install-app.sh | bash -s -- +``` + +`` needs to be the name of the application you chose in the previous step. + +This will add a new terraform module at `/infra/` and create the following CI/CD workflows: + +* `/.github/workflows/cd-.yml` +* `/.github/workflows/ci--vulnerability-scans.yml` + +### 4. Configure the application as usual + +Follow the per-application steps in [`/infra/README.md`](/infra/README.md) to configure the application. \ No newline at end of file From a3496fb990ca1b3976ba78fdd5dc8fee39b2dc1b Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 2 May 2024 12:33:46 -0700 Subject: [PATCH 43/89] Add script comments --- template-only-bin/download-and-install-app.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/template-only-bin/download-and-install-app.sh b/template-only-bin/download-and-install-app.sh index f2341b413..511801838 100755 --- a/template-only-bin/download-and-install-app.sh +++ b/template-only-bin/download-and-install-app.sh @@ -1,19 +1,17 @@ #!/usr/bin/env bash # ----------------------------------------------------------------------------- -# This script renames the application template in your project. +# This script downloads and installs infrastructure for an application. # Run this script in your project's root directory. # # Positional parameters: -# NEW_NAME (required) - the new name for the application, in either snake- or kebab-case -# OLD_NAME (optional) – the old name for the application, in either snake- or kebab-case -# Defaults to the template's short name (e.g. app-rails) +# APP_NAME (required) - the name for the application, use kebab-case # ----------------------------------------------------------------------------- set -euo pipefail APP_NAME=$1 CURRENT_VERSION=$(cat .template-version) -# Use kebab-case +# Enforce kebab-case APP_NAME_KEBAB=$(echo $APP_NAME | tr "_" "-") # Helper to get the correct sed -i behavior for both GNU sed and BSD sed (installed by default on macOS) @@ -37,6 +35,8 @@ cp -r template-infra/infra/app infra/"$APP_NAME_KEBAB" echo "Setting up new application CI/CD..." cp template-infra/.github/workflows/cd-app.yml .github/workflows/"cd-$APP_NAME_KEBAB.yml" +# This regex will capture all instances of `app` that end in a space, a double quote, a forward slash, or a hyphen +# We do this to avoid accidentally replacing the keyword `app_name` LC_ALL=C sedi "s/app\([\s\"\/\-]\)/$APP_NAME_KEBAB\1/g" .github/workflows/"cd-$APP_NAME_KEBAB.yml" cp template-infra/.github/workflows/ci-app-vulnerability-scans.yml .github/workflows/"ci-$APP_NAME_KEBAB-vulnerability-scans.yml" From 73b076c6fc453717b15a47acc688906cca7cff74 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 2 May 2024 12:33:58 -0700 Subject: [PATCH 44/89] Suppress script verbosity --- template-only-bin/download-and-install-app.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/template-only-bin/download-and-install-app.sh b/template-only-bin/download-and-install-app.sh index 511801838..2b0fa4f96 100755 --- a/template-only-bin/download-and-install-app.sh +++ b/template-only-bin/download-and-install-app.sh @@ -27,7 +27,7 @@ git clone https://github.com/navapbc/template-infra.git echo "Switching to this project's current version of the template..." cd template-infra -git checkout "$CURRENT_VERSION" +git checkout "$CURRENT_VERSION" >& /dev/null cd - >& /dev/null echo "Creating a terraform module for a new application..." From aa1b3d43a58c648a5bf75fa78cd3c11ae55f4225 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 2 May 2024 12:42:49 -0700 Subject: [PATCH 45/89] Link to new docs --- README.md | 6 +++++- template-only-docs/multiple-applications.md | 2 +- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3cfdd8ac1..3d866b319 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,11 @@ By default, the system architecture will look like this (see [system architectur ## Application Requirements -This template assumes that you have an application to deploy. See [application requirements](./template-only-docs/application-requirements.md) for more information on what is needed to use the infrastructure template. If you're using one of the [Platform application templates](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), these requirements are already met. +This template assumes that you have an application to deploy. See [application requirements](https://github.com/navapbc/template-infra/tree/main/template-only-docs/application-requirements.md) for more information on what is needed to use the infrastructure template. If you're using one of the [Platform application templates](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), these requirements are already met. + +### Multiple Applications + +This infrastructure supports deployment and CI/CD for projects with multiple applications. By default, the infrastructure assumes the project only has one application, named `app`. However, it's straightforward to include additional applications. See [how to add multiple applications](https://github.com/navapbc/template-infra/tree/main/template-only-docs/multiple-applications.md). ## Installation diff --git a/template-only-docs/multiple-applications.md b/template-only-docs/multiple-applications.md index 95ed10d3a..a434efe2e 100644 --- a/template-only-docs/multiple-applications.md +++ b/template-only-docs/multiple-applications.md @@ -1,6 +1,6 @@ # Multiple Applications -This infrastructure supports multiple deployment and CI/CD for projects with multiple applications. By default, the infrastructure assumes the project only has one application, named `app`. However, it's straightforward to include additional applications. +This infrastructure supports deployment and CI/CD for projects with multiple applications. By default, the infrastructure assumes the project only has one application, named `app`. However, it's straightforward to include additional applications. ## Prerequisites From 6263e122903c6fa240d5fad99ef04c7bb91c6805 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 2 May 2024 17:05:44 -0700 Subject: [PATCH 46/89] Update update-template.sh --- template-only-bin/update-template.sh | 71 +++++++++++++++++++++------- 1 file changed, 55 insertions(+), 16 deletions(-) diff --git a/template-only-bin/update-template.sh b/template-only-bin/update-template.sh index 2acf18b51..acd3808c9 100755 --- a/template-only-bin/update-template.sh +++ b/template-only-bin/update-template.sh @@ -1,39 +1,78 @@ -#!/bin/bash +#!/usr/bin/env bash # ----------------------------------------------------------------------------- -# This script updates template-infra in your project. Run -# This script from your project's root directory. +# This script updates template-infra in your project. +# Run this script in your project's root directory. # # Positional parameters: -# TARGET_VERSION (optional) – the version of template-infra to upgrade to. -# Defaults to main. +# APP_NAMES (required) – a comma-separated list of the apps in `/infra` with no spaces. +# Defaults to `app` +# Examples: `app`, `app,app2`, `my-app,your-app` +# +# TARGET_VERSION (optional) – the version of the template application to install. +# Defaults to main. Can be any target that can be checked out, including a branch, +# version tag, or commit hash. # ----------------------------------------------------------------------------- set -euo pipefail -TARGET_VERSION=${1:-"main"} - +APP_NAMES=${1:-"app"} +TARGET_VERSION=${2:-"main"} CURRENT_VERSION=$(cat .template-version) -echo "Clone template-infra" +echo "Cloning template-infra..." git clone https://github.com/navapbc/template-infra.git -echo "Creating patch" +echo "Creating template patch..." cd template-infra +# Checkout the version of the template to update to git checkout "$TARGET_VERSION" # Get version hash to update .template-version after patch is successful TARGET_VERSION_HASH=$(git rev-parse HEAD) # Note: Keep this list in sync with the files copied in install-template.sh -git diff "$CURRENT_VERSION" "$TARGET_VERSION" -- .github bin docs infra Makefile .dockleconfig .grype.yml .hadolint.yaml .trivyignore > update.patch -cd - +INCLUDE_PATHS=" \ + .github \ + bin \ + docs \ + infra \ + Makefile \ + .dockleconfig \ + .grype.yml \ + .hadolint.yaml \ + .trivyignore" +git diff "$CURRENT_VERSION" "$TARGET_VERSION" -- "$INCLUDE_PATHS" > patch +cd - >& /dev/null -echo "Applying patch" +echo "Applying patch..." # Note: Keep this list in sync with the removed files in install-template.sh -EXCLUDE_OPT="--exclude=.github/workflows/template-only-*" +# Exclude the app files for now. They will be handled separately below. +EXCLUDE_OPT=" \ + --exclude=.github/workflows/template-only-* \ + --exclude=.github/workflows/*-app-*.yml \ + --exclude=infra/app" git apply "$EXCLUDE_OPT" --allow-empty template-infra/update.patch -echo "Saving new template version to .template-infra" +# Loop through the comma-separated list of apps +for APP_NAME in ${APP_NAMES//,/ } +do + echo "Creating patch for $APP_NAME..." + # This creates a git patch comparing a project app with the upstream `/infra/app` directory + # --no-index allows us to compare differently named directories + # --dst-prefix="" removes the destination prefix, essentially having the effect of removing the additional `template-infra/` + # path prefix, making the patch easily appliable + # || true is necessary because this bash script includes `set -e`, which will immediately exit for any non-zero exit codes + # That's generally correct, but `git diff --no-index` will return 1 to indicate differences between the files. That's correct + # and expected behavior. + git diff --no-index --dst-prefix="" "infra/$APP_NAME" template-infra/infra/app > "template-infra/$APP_NAME.patch" || true + + echo "Applying patch for $APP_NAME..." + git apply --allow-empty "template-infra/$APP_NAME.patch" +done + +echo "Saving new template version to .template-infra..." echo "$TARGET_VERSION_HASH" > .template-version -echo "Clean up template-infra folder" -rm -fr template-infra +echo "Cleaning up template-infra folder..." +rm -rf template-infra + +echo "...Done." From 8f3ec62915e28c1bf3fd4ec340c110d17ffddff2 Mon Sep 17 00:00:00 2001 From: Rocket Date: Fri, 3 May 2024 14:38:19 -0700 Subject: [PATCH 47/89] Add WIP --- template-only-bin/update-template.sh | 210 ++++++++++++++++++++++----- 1 file changed, 176 insertions(+), 34 deletions(-) diff --git a/template-only-bin/update-template.sh b/template-only-bin/update-template.sh index acd3808c9..2c4450e16 100755 --- a/template-only-bin/update-template.sh +++ b/template-only-bin/update-template.sh @@ -14,47 +14,172 @@ # ----------------------------------------------------------------------------- set -euo pipefail -APP_NAMES=${1:-"app"} +APP_NAMES=$1 TARGET_VERSION=${2:-"main"} CURRENT_VERSION=$(cat .template-version) -echo "Cloning template-infra..." -git clone https://github.com/navapbc/template-infra.git +echo "=====================================================================" +echo "Updating template-infra" +echo "=====================================================================" +echo "APP_NAMES=$APP_NAMES" +echo "CURRENT_VERSION=$CURRENT_VERSION" +echo "TARGET_VERSION=$TARGET_VERSION" +echo -echo "Creating template patch..." -cd template-infra -# Checkout the version of the template to update to -git checkout "$TARGET_VERSION" +# @TODO verify that you can pass in empty string and things will update ok +# Verify that all the apps passed in exist +# Loop through the comma-separated list of apps +for APP_NAME in ${APP_NAMES//,/ } +do + if [ ! -d "infra/$APP_NAME" ]; then + echo "Error: infra/$APP_NAME does not exist. Exiting." + exit 1 + fi +done + +echo "---------------------------------------------------------------------" +echo "1. Patching: main template" +echo "---------------------------------------------------------------------" +echo "Temporarily setting template as a remote 'upstream-template-infra'..." +git remote add upstream-template-infra https://github.com/navapbc/template-infra.git + +echo +echo "---------------------------------------------------------------------" +echo "Fetching target version: $TARGET_VERSION" +echo "---------------------------------------------------------------------" +git fetch upstream-template-infra "$TARGET_VERSION" # Get version hash to update .template-version after patch is successful -TARGET_VERSION_HASH=$(git rev-parse HEAD) +TARGET_VERSION_HASH=$(git rev-parse "upstream-template-infra/$TARGET_VERSION") +echo +echo "---------------------------------------------------------------------" +echo "Creating template patch" +echo "---------------------------------------------------------------------" # Note: Keep this list in sync with the files copied in install-template.sh -INCLUDE_PATHS=" \ - .github \ - bin \ - docs \ - infra \ - Makefile \ - .dockleconfig \ - .grype.yml \ - .hadolint.yaml \ - .trivyignore" -git diff "$CURRENT_VERSION" "$TARGET_VERSION" -- "$INCLUDE_PATHS" > patch -cd - >& /dev/null - -echo "Applying patch..." -# Note: Keep this list in sync with the removed files in install-template.sh -# Exclude the app files for now. They will be handled separately below. -EXCLUDE_OPT=" \ - --exclude=.github/workflows/template-only-* \ - --exclude=.github/workflows/*-app-*.yml \ - --exclude=infra/app" -git apply "$EXCLUDE_OPT" --allow-empty template-infra/update.patch +INCLUDE_PATHS=".github bin docs infra Makefile .dockleconfig .grype.yml .hadolint.yaml .trivyignore" +# Note: Exclude template-only files, terraform deployment files, and all files related to +# application(s) as those are handled separately below. +EXCLUDE_PATHS=" + ':!.github/workflows/template-only-*' + ':!*.terraform*' + ':!*.tfbackend' + ':!.github/workflows/*app*.yml'" -# Loop through the comma-separated list of apps +# Ignore all applications to be updated for APP_NAME in ${APP_NAMES//,/ } do + EXCLUDE_PATHS="${EXCLUDE_PATHS} ':!infra/$APP_NAME'" +done + +STAT_COMMAND="git --no-pager diff -R --stat upstream-template-infra/rocket/remove-tf-lock -- $(echo $INCLUDE_PATHS) $(echo $EXCLUDE_PATHS)" +eval "$STAT_COMMAND" + +DIFF_COMMAND="git diff -R upstream-template-infra/rocket/remove-tf-lock -- $(echo $INCLUDE_PATHS) $(echo $EXCLUDE_PATHS)" +eval "$DIFF_COMMAND > main-template.patch" + +echo +echo "---------------------------------------------------------------------" +echo "Applying template patch" +echo "---------------------------------------------------------------------" +echo "Applying..." +git apply --allow-empty main-template.patch + +echo +echo "---------------------------------------------------------------------" +echo "Cleaning up" +echo "---------------------------------------------------------------------" +echo "Removing patch file..." +rm main-template.patch + +echo "Removing git remote..." +git remote rm upstream-template-infra + + +# -------------------------------------------------------------------------------------------- +# echo "---------------------------------------------------------------------" +# echo "Temporarily cloning template-infra" +# echo "---------------------------------------------------------------------" +# git clone https://github.com/navapbc/template-infra.git + +# echo +# echo "---------------------------------------------------------------------" +# echo "Checking out target version: $TARGET_VERSION" +# echo "---------------------------------------------------------------------" +# cd template-infra +# # Checkout the version of the template to update to +# git checkout "$TARGET_VERSION" + +# # Get version hash to update .template-version after patch is successful +# TARGET_VERSION_HASH=$(git rev-parse HEAD) + +# echo +# echo "---------------------------------------------------------------------" +# echo "Patching: main template" +# echo "---------------------------------------------------------------------" +# echo "Creating template patch..." +# # Note: Keep this list in sync with the files copied in install-template.sh +# INCLUDE_PATHS=" +# .github +# bin +# docs +# infra +# Makefile +# .dockleconfig +# .grype.yml +# .hadolint.yaml +# .trivyignore" +# echo "$INCLUDE_PATHS" + +# EXCLUDE_PATHS=" +# .github/workflows/template-only-* +# infra/app +# *.terraform* +# " +# MY_PATHS=".github bin docs infra Makefile .dockleconfig .grype.yml .hadolint.yaml .trivyignore" +# # git diff "$CURRENT_VERSION" "$TARGET_VERSION" -- "$INCLUDE_PATHS" > main-template.patch +# # git diff "$CURRENT_VERSION" "$TARGET_VERSION" -- "$MY_PATHS" > main-template.patch +# git diff --no-index . template-infra +# cd - >& /dev/null + +# echo "Applying template patch..." +# # Note: Keep this list in sync with the removed files in install-template.sh +# # Exclude the app files for now. They will be handled separately below. +# # @TODO include of this backwards compatibility exclude, we should shift the strategy to git diff --no-index +# # EXCLUDE_OPT=" \ +# # --exclude=.github/workflows/template-only-* \ +# # --exclude=infra/app \ +# # --exclude=infra/app/build-repository/.terraform.lock.hcl" +# # git apply $EXCLUDE_OPT --allow-empty template-infra/main-template.patch + +# -------------------------------------------------------------------------------------------- + +echo "---------------------------------------------------------------------" +echo "2. Prepare to patch application(s)" +echo "---------------------------------------------------------------------" +# Empty step to have a nice header +echo "Preparing..." + +echo "---------------------------------------------------------------------" +echo "Temporarily cloning template-infra" +echo "---------------------------------------------------------------------" +git clone https://github.com/navapbc/template-infra.git + +echo +echo "---------------------------------------------------------------------" +echo "Checking out target version: $TARGET_VERSION" +echo "---------------------------------------------------------------------" +cd template-infra +git checkout "$TARGET_VERSION" + +# Patch each application +STEP_COUNT=3 +for APP_NAME in ${APP_NAMES//,/ } +do + echo + echo "---------------------------------------------------------------------" + echo " $STEP_COUNT. Patching application: $APP_NAME" + echo "---------------------------------------------------------------------" echo "Creating patch for $APP_NAME..." # This creates a git patch comparing a project app with the upstream `/infra/app` directory # --no-index allows us to compare differently named directories @@ -66,13 +191,30 @@ do git diff --no-index --dst-prefix="" "infra/$APP_NAME" template-infra/infra/app > "template-infra/$APP_NAME.patch" || true echo "Applying patch for $APP_NAME..." - git apply --allow-empty "template-infra/$APP_NAME.patch" + # This is the second part that allows comparing between directories with different names + # -p3 strips the first 3 path fragments from filenames + # Ex: if the filename is `a/infra/app/app-config/dev.tf`, then -p3 causes it to become: `app-config/dev.tf` + # --directory="infra/$APP_NAME" prepends path parts to the filename + # Ex: if the filename is `a/infra/app/app-config/dev.tf`, then --directory causes it to become: `/infra/$APP_NAME/app-config/dev.tf` + git apply -p3 --directory="infra/$APP_NAME" --allow-empty "template-infra/$APP_NAME.patch" + + # Increment step counter + STEP_COUNT=$((STEP_COUNT+1)) + + # @TODO patch for CI/CD done +echo +echo "---------------------------------------------------------------------" +echo "Cleaning up" +echo "---------------------------------------------------------------------" echo "Saving new template version to .template-infra..." echo "$TARGET_VERSION_HASH" > .template-version -echo "Cleaning up template-infra folder..." -rm -rf template-infra +# echo "Cleaning up template-infra folder..." +# rm -rf template-infra -echo "...Done." +echo +echo "=====================================================================" +echo "Done." +echo "=====================================================================" From 9f64936a88fc7a0fb16c1cfb439e63feddd8d67a Mon Sep 17 00:00:00 2001 From: Rocket Date: Fri, 3 May 2024 15:59:40 -0700 Subject: [PATCH 48/89] WIP update --- template-only-bin/update-template.sh | 129 ++++++--------------------- 1 file changed, 29 insertions(+), 100 deletions(-) diff --git a/template-only-bin/update-template.sh b/template-only-bin/update-template.sh index 2c4450e16..661db79c4 100755 --- a/template-only-bin/update-template.sh +++ b/template-only-bin/update-template.sh @@ -26,8 +26,17 @@ echo "CURRENT_VERSION=$CURRENT_VERSION" echo "TARGET_VERSION=$TARGET_VERSION" echo -# @TODO verify that you can pass in empty string and things will update ok -# Verify that all the apps passed in exist +# Check: that $APP_NAMES is not empty string +if [ -z "$APP_NAMES" ]; then + echo "Error: The first argument (APP_NAMES) cannot be empty." + echo " Please supply a comma-separated list of applications in /infra." + echo " Example: app" + echo " Example: app,app2" + echo "Exiting." + exit 1 +fi + +# Check: that all the apps passed in exist # Loop through the comma-separated list of apps for APP_NAME in ${APP_NAMES//,/ } do @@ -41,21 +50,14 @@ echo "---------------------------------------------------------------------" echo "1. Patching: main template" echo "---------------------------------------------------------------------" echo "Temporarily setting template as a remote 'upstream-template-infra'..." +echo git remote add upstream-template-infra https://github.com/navapbc/template-infra.git -echo -echo "---------------------------------------------------------------------" -echo "Fetching target version: $TARGET_VERSION" -echo "---------------------------------------------------------------------" git fetch upstream-template-infra "$TARGET_VERSION" # Get version hash to update .template-version after patch is successful TARGET_VERSION_HASH=$(git rev-parse "upstream-template-infra/$TARGET_VERSION") -echo -echo "---------------------------------------------------------------------" -echo "Creating template patch" -echo "---------------------------------------------------------------------" # Note: Keep this list in sync with the files copied in install-template.sh INCLUDE_PATHS=".github bin docs infra Makefile .dockleconfig .grype.yml .hadolint.yaml .trivyignore" # Note: Exclude template-only files, terraform deployment files, and all files related to @@ -78,99 +80,17 @@ eval "$STAT_COMMAND" DIFF_COMMAND="git diff -R upstream-template-infra/rocket/remove-tf-lock -- $(echo $INCLUDE_PATHS) $(echo $EXCLUDE_PATHS)" eval "$DIFF_COMMAND > main-template.patch" -echo -echo "---------------------------------------------------------------------" -echo "Applying template patch" -echo "---------------------------------------------------------------------" -echo "Applying..." +# Actually apply the patch git apply --allow-empty main-template.patch echo echo "---------------------------------------------------------------------" -echo "Cleaning up" -echo "---------------------------------------------------------------------" -echo "Removing patch file..." -rm main-template.patch - -echo "Removing git remote..." -git remote rm upstream-template-infra - - -# -------------------------------------------------------------------------------------------- -# echo "---------------------------------------------------------------------" -# echo "Temporarily cloning template-infra" -# echo "---------------------------------------------------------------------" -# git clone https://github.com/navapbc/template-infra.git - -# echo -# echo "---------------------------------------------------------------------" -# echo "Checking out target version: $TARGET_VERSION" -# echo "---------------------------------------------------------------------" -# cd template-infra -# # Checkout the version of the template to update to -# git checkout "$TARGET_VERSION" - -# # Get version hash to update .template-version after patch is successful -# TARGET_VERSION_HASH=$(git rev-parse HEAD) - -# echo -# echo "---------------------------------------------------------------------" -# echo "Patching: main template" -# echo "---------------------------------------------------------------------" -# echo "Creating template patch..." -# # Note: Keep this list in sync with the files copied in install-template.sh -# INCLUDE_PATHS=" -# .github -# bin -# docs -# infra -# Makefile -# .dockleconfig -# .grype.yml -# .hadolint.yaml -# .trivyignore" -# echo "$INCLUDE_PATHS" - -# EXCLUDE_PATHS=" -# .github/workflows/template-only-* -# infra/app -# *.terraform* -# " -# MY_PATHS=".github bin docs infra Makefile .dockleconfig .grype.yml .hadolint.yaml .trivyignore" -# # git diff "$CURRENT_VERSION" "$TARGET_VERSION" -- "$INCLUDE_PATHS" > main-template.patch -# # git diff "$CURRENT_VERSION" "$TARGET_VERSION" -- "$MY_PATHS" > main-template.patch -# git diff --no-index . template-infra -# cd - >& /dev/null - -# echo "Applying template patch..." -# # Note: Keep this list in sync with the removed files in install-template.sh -# # Exclude the app files for now. They will be handled separately below. -# # @TODO include of this backwards compatibility exclude, we should shift the strategy to git diff --no-index -# # EXCLUDE_OPT=" \ -# # --exclude=.github/workflows/template-only-* \ -# # --exclude=infra/app \ -# # --exclude=infra/app/build-repository/.terraform.lock.hcl" -# # git apply $EXCLUDE_OPT --allow-empty template-infra/main-template.patch - -# -------------------------------------------------------------------------------------------- - -echo "---------------------------------------------------------------------" -echo "2. Prepare to patch application(s)" -echo "---------------------------------------------------------------------" -# Empty step to have a nice header -echo "Preparing..." - -echo "---------------------------------------------------------------------" -echo "Temporarily cloning template-infra" +echo "2. Preparing to patch application(s)" echo "---------------------------------------------------------------------" git clone https://github.com/navapbc/template-infra.git - -echo -echo "---------------------------------------------------------------------" -echo "Checking out target version: $TARGET_VERSION" -echo "---------------------------------------------------------------------" cd template-infra git checkout "$TARGET_VERSION" +cd - >& /dev/null # Patch each application STEP_COUNT=3 @@ -178,9 +98,8 @@ for APP_NAME in ${APP_NAMES//,/ } do echo echo "---------------------------------------------------------------------" - echo " $STEP_COUNT. Patching application: $APP_NAME" + echo "$STEP_COUNT. Patching application: $APP_NAME" echo "---------------------------------------------------------------------" - echo "Creating patch for $APP_NAME..." # This creates a git patch comparing a project app with the upstream `/infra/app` directory # --no-index allows us to compare differently named directories # --dst-prefix="" removes the destination prefix, essentially having the effect of removing the additional `template-infra/` @@ -190,13 +109,17 @@ do # and expected behavior. git diff --no-index --dst-prefix="" "infra/$APP_NAME" template-infra/infra/app > "template-infra/$APP_NAME.patch" || true - echo "Applying patch for $APP_NAME..." # This is the second part that allows comparing between directories with different names # -p3 strips the first 3 path fragments from filenames # Ex: if the filename is `a/infra/app/app-config/dev.tf`, then -p3 causes it to become: `app-config/dev.tf` # --directory="infra/$APP_NAME" prepends path parts to the filename # Ex: if the filename is `a/infra/app/app-config/dev.tf`, then --directory causes it to become: `/infra/$APP_NAME/app-config/dev.tf` - git apply -p3 --directory="infra/$APP_NAME" --allow-empty "template-infra/$APP_NAME.patch" + # This is the stat version of the command to output the changes + STAT_COMMAND="git --no-pager apply --stat -p3 --directory=infra/$APP_NAME --allow-empty template-infra/$APP_NAME.patch --exclude='*.tfbackend' --exclude='*.terraform*'" + eval "$STAT_COMMAND" + + # Actually running the command `git apply` + git apply -p3 --directory="infra/$APP_NAME" --allow-empty "template-infra/$APP_NAME.patch" --exclude="*.tfbackend" --exclude="*.terraform*" # Increment step counter STEP_COUNT=$((STEP_COUNT+1)) @@ -206,11 +129,17 @@ done echo echo "---------------------------------------------------------------------" -echo "Cleaning up" +echo "$STEP_COUNT. Cleaning up" echo "---------------------------------------------------------------------" echo "Saving new template version to .template-infra..." echo "$TARGET_VERSION_HASH" > .template-version +echo "Removing patch files..." +rm main-template.patch + +echo "Removing git remote..." +git remote rm upstream-template-infra + # echo "Cleaning up template-infra folder..." # rm -rf template-infra From 070ac8f34faa450ffcd43be22e3b90da7092cee9 Mon Sep 17 00:00:00 2001 From: Rocket Date: Fri, 3 May 2024 17:06:11 -0700 Subject: [PATCH 49/89] Fix target types --- template-only-bin/update-template.sh | 73 ++++++++++++++++++++++------ 1 file changed, 58 insertions(+), 15 deletions(-) diff --git a/template-only-bin/update-template.sh b/template-only-bin/update-template.sh index 661db79c4..72ada80b1 100755 --- a/template-only-bin/update-template.sh +++ b/template-only-bin/update-template.sh @@ -9,26 +9,29 @@ # Examples: `app`, `app,app2`, `my-app,your-app` # # TARGET_VERSION (optional) – the version of the template application to install. -# Defaults to main. Can be any target that can be checked out, including a branch, -# version tag, or commit hash. +# Defaults to main. Can be a branch, commit hash, or tag. +# +# TARGET_VERSION_TYPE (optional) – the version of the template application to install. +# Defaults to branch. Can be: branch, commit, tag. # ----------------------------------------------------------------------------- set -euo pipefail APP_NAMES=$1 TARGET_VERSION=${2:-"main"} +TARGET_VERSION_TYPE=${3:-"branch"} CURRENT_VERSION=$(cat .template-version) +TARGET_VERSION_HASH="" echo "=====================================================================" echo "Updating template-infra" echo "=====================================================================" echo "APP_NAMES=$APP_NAMES" -echo "CURRENT_VERSION=$CURRENT_VERSION" echo "TARGET_VERSION=$TARGET_VERSION" -echo +echo "TARGET_VERSION_TYPE=$TARGET_VERSION_TYPE" # Check: that $APP_NAMES is not empty string if [ -z "$APP_NAMES" ]; then - echo "Error: The first argument (APP_NAMES) cannot be empty." + echo "Error: APP_NAMES cannot be empty." echo " Please supply a comma-separated list of applications in /infra." echo " Example: app" echo " Example: app,app2" @@ -46,20 +49,58 @@ do fi done +# Check: that TARGET_VERSION_TYPE is valid +case $TARGET_VERSION_TYPE in + "branch"|"commit"|"tag") + # Acceptable options, do nothing + ;; + *) + echo "Error: TARGET_VERSION_TYPE must be: branch, commit, or tag" + exit 1 +esac + +echo echo "---------------------------------------------------------------------" echo "1. Patching: main template" echo "---------------------------------------------------------------------" -echo "Temporarily setting template as a remote 'upstream-template-infra'..." +echo "Temporarily creating remote 'upstream-template-infra'..." echo git remote add upstream-template-infra https://github.com/navapbc/template-infra.git -git fetch upstream-template-infra "$TARGET_VERSION" +echo "Fetching from upstream remote..." +echo +git fetch upstream-template-infra -# Get version hash to update .template-version after patch is successful -TARGET_VERSION_HASH=$(git rev-parse "upstream-template-infra/$TARGET_VERSION") +# Get target version hash +echo +echo "Converting $TARGET_VERSION to hash..." +case $TARGET_VERSION_TYPE in + "branch") + TARGET_VERSION_HASH=$(git rev-parse upstream-template-infra/$TARGET_VERSION) + ;; + "commit") + echo "No conversion needed." + TARGET_VERSION_HASH=$TARGET_VERSION + ;; + "tag") + TARGET_VERSION_HASH=$(git ls-remote --tags upstream-template-infra $TARGET_VERSION | cut -d$'\t' -f1) + ;; +esac +echo "TARGET_VERSION_HASH=$TARGET_VERSION_HASH" +echo # Note: Keep this list in sync with the files copied in install-template.sh -INCLUDE_PATHS=".github bin docs infra Makefile .dockleconfig .grype.yml .hadolint.yaml .trivyignore" +INCLUDE_PATHS=" + .github + bin + docs + infra + Makefile + .dockleconfig + .grype.yml + .hadolint.yaml + .trivyignore" + # Note: Exclude template-only files, terraform deployment files, and all files related to # application(s) as those are handled separately below. EXCLUDE_PATHS=" @@ -68,19 +109,21 @@ EXCLUDE_PATHS=" ':!*.tfbackend' ':!.github/workflows/*app*.yml'" -# Ignore all applications to be updated +# Exclude all applications for APP_NAME in ${APP_NAMES//,/ } do EXCLUDE_PATHS="${EXCLUDE_PATHS} ':!infra/$APP_NAME'" done -STAT_COMMAND="git --no-pager diff -R --stat upstream-template-infra/rocket/remove-tf-lock -- $(echo $INCLUDE_PATHS) $(echo $EXCLUDE_PATHS)" +# Show the changes to be made +STAT_COMMAND="git --no-pager diff -R --stat $TARGET_VERSION_HASH -- $(echo $INCLUDE_PATHS) $(echo $EXCLUDE_PATHS)" eval "$STAT_COMMAND" -DIFF_COMMAND="git diff -R upstream-template-infra/rocket/remove-tf-lock -- $(echo $INCLUDE_PATHS) $(echo $EXCLUDE_PATHS)" +# Make the patch file +DIFF_COMMAND="git diff -R $TARGET_VERSION_HASH -- $(echo $INCLUDE_PATHS) $(echo $EXCLUDE_PATHS)" eval "$DIFF_COMMAND > main-template.patch" -# Actually apply the patch +# Apply the patch file git apply --allow-empty main-template.patch echo @@ -89,7 +132,7 @@ echo "2. Preparing to patch application(s)" echo "---------------------------------------------------------------------" git clone https://github.com/navapbc/template-infra.git cd template-infra -git checkout "$TARGET_VERSION" +git checkout "$TARGET_VERSION_HASH" cd - >& /dev/null # Patch each application From 08917f5c0200875898d9e183009825360df18bb0 Mon Sep 17 00:00:00 2001 From: Rocket Date: Fri, 3 May 2024 17:14:51 -0700 Subject: [PATCH 50/89] Add more documentation --- template-only-bin/update-template.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/template-only-bin/update-template.sh b/template-only-bin/update-template.sh index 72ada80b1..0a2adcec4 100755 --- a/template-only-bin/update-template.sh +++ b/template-only-bin/update-template.sh @@ -3,6 +3,17 @@ # This script updates template-infra in your project. # Run this script in your project's root directory. # +# This script uses git to create a patch files between the current HEAD and the +# TARGET_VERSION argument. For the main template, it sets the upstream github repo as a +# remote, creates a patch file and applies it. The git-apply excludes template-only +# files. Because applications often do not retain the default `app` name, a different +# approach is needed. To create the patch for applications, the upstream repo is cloned +# into a sub-directory and `git diff --no-index` is used to compare differently-named +# directories. +# +# Usage: +# ./template-only-bin/update-template.sh +# # Positional parameters: # APP_NAMES (required) – a comma-separated list of the apps in `/infra` with no spaces. # Defaults to `app` @@ -13,6 +24,14 @@ # # TARGET_VERSION_TYPE (optional) – the version of the template application to install. # Defaults to branch. Can be: branch, commit, tag. +# +# Examples: +# - To update a project with one application named `app` to the latest upstream `main`: +# ./template-only-bin/update-template.sh app +# - To update a project with two applications to a specific commit: +# ./template-only-bin/update-template.sh app,app2 d42963d007e55cc37ef666019428b1d06a25cf71 commit +# - To update a project with three applications to a tag: +# ./template-only-bin/update-template.sh alpha,beta,gamma-three v0.8.0 tag # ----------------------------------------------------------------------------- set -euo pipefail From 831ed9eb0da82575b6c042202f7d552fdc7c81f3 Mon Sep 17 00:00:00 2001 From: Rocket Date: Fri, 3 May 2024 18:56:23 -0700 Subject: [PATCH 51/89] Refactor. Update docs --- README.md | 38 ++++-- template-only-bin/download-and-install-app.sh | 21 +--- template-only-bin/install-app.sh | 32 +++++ template-only-bin/update-template.sh | 111 ++++++++++-------- 4 files changed, 123 insertions(+), 79 deletions(-) create mode 100644 template-only-bin/install-app.sh diff --git a/README.md b/README.md index 3d866b319..fcec2ac99 100644 --- a/README.md +++ b/README.md @@ -45,24 +45,36 @@ After downloading and installing the template into your project: ## Updates -There are multiple ways to receive template updates on your project. For most updates, you can simply run the [update-template.sh](/template-only-bin/update-template.sh) script +To apply template updates to your project, run ```bash -curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s +curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- ``` -If the update fails the simplest option may be to re-run the installation script above and manually review the changes. + is a required argument. It must be a comma-separated list (no spaces) of the apps in `/infra`. App names are expected to be hyphen-separated (i.e. kebab-case). + Examples: `app`, `app,app2`, `my-app,your-app` -**Remember:** Make sure to read the release notes in case there are breaking changes you need to address. - -### Renamed applications - -If you renamed your application from `infra/app` to something else like `infra/foo`, then first rename your app back to `infra/app` before applying the updates e.g. +By default, the update script will update to the latest commit on the `main` branch in the template repo. If you want to update to a different branch, a specific commit, or a specific tag (e.g. a release tag), run this instead ```bash -mv foo app -mv infra/foo infra/app -curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -mv infra/app infra/foo -mv app foo +curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- ``` + should be the version of the template to install. This can be a branch, commit hash, or tag. + should be the type of provided. Defaults to `branch`. This can be: `branch`, `commit`, or `tag`. + +Examples: +- To update a project with one application named `app` to `main` in the template repo: + ```bash + curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- app + ``` +- To update a project with two applications to a specific commit: + ```bash + curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- app,app2 d42963d007e55cc37ef666019428b1d06a25cf71 commit + ``` + +- To update a project with three applications to a tag: + ```bash + curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- alpha,beta,gamma-three v0.8.0 tag + ``` + +**Remember:** Make sure to read the release notes in case there are breaking changes you need to address. diff --git a/template-only-bin/download-and-install-app.sh b/template-only-bin/download-and-install-app.sh index 2b0fa4f96..680934e09 100755 --- a/template-only-bin/download-and-install-app.sh +++ b/template-only-bin/download-and-install-app.sh @@ -14,14 +14,6 @@ CURRENT_VERSION=$(cat .template-version) # Enforce kebab-case APP_NAME_KEBAB=$(echo $APP_NAME | tr "_" "-") -# Helper to get the correct sed -i behavior for both GNU sed and BSD sed (installed by default on macOS) -# Hat tip: https://stackoverflow.com/a/38595160 -sedi () { - sed --version >/dev/null 2>&1 && sed -i -- "$@" || sed -i "" "$@" -} -# Export the function so it can be used later on -export -f sedi - echo "Cloning template-infra..." git clone https://github.com/navapbc/template-infra.git @@ -30,17 +22,8 @@ cd template-infra git checkout "$CURRENT_VERSION" >& /dev/null cd - >& /dev/null -echo "Creating a terraform module for a new application..." -cp -r template-infra/infra/app infra/"$APP_NAME_KEBAB" - -echo "Setting up new application CI/CD..." -cp template-infra/.github/workflows/cd-app.yml .github/workflows/"cd-$APP_NAME_KEBAB.yml" -# This regex will capture all instances of `app` that end in a space, a double quote, a forward slash, or a hyphen -# We do this to avoid accidentally replacing the keyword `app_name` -LC_ALL=C sedi "s/app\([\s\"\/\-]\)/$APP_NAME_KEBAB\1/g" .github/workflows/"cd-$APP_NAME_KEBAB.yml" - -cp template-infra/.github/workflows/ci-app-vulnerability-scans.yml .github/workflows/"ci-$APP_NAME_KEBAB-vulnerability-scans.yml" -LC_ALL=C sedi "s/app\([\s\"\/\-]\)/$APP_NAME_KEBAB\1/g" .github/workflows/"ci-$APP_NAME_KEBAB-vulnerability-scans.yml" +# Install the app +curl "https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/install-app.sh" | bash -s -- $APP_NAME echo "Cleaning up template-infra folder..." rm -fr template-infra diff --git a/template-only-bin/install-app.sh b/template-only-bin/install-app.sh new file mode 100644 index 000000000..18d027fc9 --- /dev/null +++ b/template-only-bin/install-app.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +# ----------------------------------------------------------------------------- +# This script downloads and installs infrastructure for an application. +# Run this script in your project's root directory. +# +# Positional parameters: +# APP_NAME (required) - the name for the application, use kebab-case +# ----------------------------------------------------------------------------- +set -euo pipefail + +APP_NAME=$1 +DST_PREFIX=${2:-""} + +echo "Creating a terraform module for application: $APP_NAME..." +cp -r template-infra/infra/app "${DST_PREFIX}infra/$APP_NAME" + +# Helper to get the correct sed -i behavior for both GNU sed and BSD sed (installed by default on macOS) +# Hat tip: https://stackoverflow.com/a/38595160 +sedi () { + sed --version >/dev/null 2>&1 && sed -i -- "$@" || sed -i "" "$@" +} +# Export the function so it can be used below +export -f sedi + +echo "Setting up CI/CD for application: $APP_NAME..." +cp template-infra/.github/workflows/cd-app.yml "${DST_PREFIX}.github/workflows/cd-$APP_NAME.yml" +# This regex will capture all instances of `app` that end in a space, a double quote, a forward slash, or a hyphen +# We do this to avoid accidentally replacing the keyword `app_name` +LC_ALL=C sedi "s/app\([\s\"\/\-]\)/$APP_NAME\1/g" "${DST_PREFIX}.github/workflows/cd-$APP_NAME.yml" + +cp template-infra/.github/workflows/ci-app-vulnerability-scans.yml "${DST_PREFIX}.github/workflows/ci-$APP_NAME-vulnerability-scans.yml" +LC_ALL=C sedi "s/app\([\s\"\/\-]\)/$APP_NAME\1/g" "${DST_PREFIX}.github/workflows/ci-$APP_NAME-vulnerability-scans.yml" diff --git a/template-only-bin/update-template.sh b/template-only-bin/update-template.sh index 0a2adcec4..fdb78f750 100755 --- a/template-only-bin/update-template.sh +++ b/template-only-bin/update-template.sh @@ -3,33 +3,35 @@ # This script updates template-infra in your project. # Run this script in your project's root directory. # -# This script uses git to create a patch files between the current HEAD and the -# TARGET_VERSION argument. For the main template, it sets the upstream github repo as a -# remote, creates a patch file and applies it. The git-apply excludes template-only -# files. Because applications often do not retain the default `app` name, a different -# approach is needed. To create the patch for applications, the upstream repo is cloned -# into a sub-directory and `git diff --no-index` is used to compare differently-named -# directories. +# This script uses git to create patch files between the current HEAD and the +# TARGET_VERSION argument. For the main portion of the template, it sets the upstream +# github repo as a remote, creates a patch file, and applies it. The git-apply excludes +# template-only files. For applications, because they often do not retain the default +# `app` name, a different approach is needed. To create the patch for applications, the +# upstream repo is cloned into a sub-directory and `git diff --no-index` is used to +# compare differently-named directories. # # Usage: # ./template-only-bin/update-template.sh # # Positional parameters: -# APP_NAMES (required) – a comma-separated list of the apps in `/infra` with no spaces. -# Defaults to `app` +# APP_NAMES (required) – a comma-separated list (no spaces) of the apps in `/infra`. App +# names are expected to be hyphen-separated (i.e. kebab-case). # Examples: `app`, `app,app2`, `my-app,your-app` # -# TARGET_VERSION (optional) – the version of the template application to install. +# TARGET_VERSION (optional) – the version of the template to install # Defaults to main. Can be a branch, commit hash, or tag. # -# TARGET_VERSION_TYPE (optional) – the version of the template application to install. -# Defaults to branch. Can be: branch, commit, tag. +# TARGET_VERSION_TYPE (optional) – the type of TARGET_VERSION provided +# Defaults to branch. Can be: branch, commit, or tag. # # Examples: -# - To update a project with one application named `app` to the latest upstream `main`: +# - To update a project with one application named `app` to `main` in the template repo: # ./template-only-bin/update-template.sh app +# # - To update a project with two applications to a specific commit: # ./template-only-bin/update-template.sh app,app2 d42963d007e55cc37ef666019428b1d06a25cf71 commit +# # - To update a project with three applications to a tag: # ./template-only-bin/update-template.sh alpha,beta,gamma-three v0.8.0 tag # ----------------------------------------------------------------------------- @@ -47,8 +49,15 @@ echo "=====================================================================" echo "APP_NAMES=$APP_NAMES" echo "TARGET_VERSION=$TARGET_VERSION" echo "TARGET_VERSION_TYPE=$TARGET_VERSION_TYPE" +echo + +# Check: that the git repo is clean +if [ -n "$(git status --porcelain)" ]; then + echo "Error: Commit or stash all changes before proceeding. Exiting." + exit 1 +fi -# Check: that $APP_NAMES is not empty string +# Check: that $APP_NAMES is not an empty string if [ -z "$APP_NAMES" ]; then echo "Error: APP_NAMES cannot be empty." echo " Please supply a comma-separated list of applications in /infra." @@ -58,7 +67,7 @@ if [ -z "$APP_NAMES" ]; then exit 1 fi -# Check: that all the apps passed in exist +# Check: that all the apps exist # Loop through the comma-separated list of apps for APP_NAME in ${APP_NAMES//,/ } do @@ -74,11 +83,10 @@ case $TARGET_VERSION_TYPE in # Acceptable options, do nothing ;; *) - echo "Error: TARGET_VERSION_TYPE must be: branch, commit, or tag" + echo "Error: TARGET_VERSION_TYPE must be: branch, commit, or tag. Exiting." exit 1 esac -echo echo "---------------------------------------------------------------------" echo "1. Patching: main template" echo "---------------------------------------------------------------------" @@ -88,21 +96,20 @@ git remote add upstream-template-infra https://github.com/navapbc/template-infra echo "Fetching from upstream remote..." echo -git fetch upstream-template-infra +git fetch upstream-template-infra >& /dev/null # Get target version hash -echo echo "Converting $TARGET_VERSION to hash..." case $TARGET_VERSION_TYPE in "branch") - TARGET_VERSION_HASH=$(git rev-parse upstream-template-infra/$TARGET_VERSION) + TARGET_VERSION_HASH="$(git rev-parse upstream-template-infra/$TARGET_VERSION)" ;; "commit") echo "No conversion needed." TARGET_VERSION_HASH=$TARGET_VERSION ;; "tag") - TARGET_VERSION_HASH=$(git ls-remote --tags upstream-template-infra $TARGET_VERSION | cut -d$'\t' -f1) + TARGET_VERSION_HASH="$(git ls-remote --tags upstream-template-infra $TARGET_VERSION | cut -d$'\t' -f1)" ;; esac echo "TARGET_VERSION_HASH=$TARGET_VERSION_HASH" @@ -120,13 +127,12 @@ INCLUDE_PATHS=" .hadolint.yaml .trivyignore" -# Note: Exclude template-only files, terraform deployment files, and all files related to -# application(s) as those are handled separately below. +# Note: Exclude terraform deployment files, and CI/CD workflows as those are handled +# separately below. EXCLUDE_PATHS=" - ':!.github/workflows/template-only-*' ':!*.terraform*' ':!*.tfbackend' - ':!.github/workflows/*app*.yml'" + ':!.github/workflows/'" # Exclude all applications for APP_NAME in ${APP_NAMES//,/ } @@ -162,33 +168,43 @@ do echo "---------------------------------------------------------------------" echo "$STEP_COUNT. Patching application: $APP_NAME" echo "---------------------------------------------------------------------" - # This creates a git patch comparing a project app with the upstream `/infra/app` directory - # --no-index allows us to compare differently named directories - # --dst-prefix="" removes the destination prefix, essentially having the effect of removing the additional `template-infra/` - # path prefix, making the patch easily appliable - # || true is necessary because this bash script includes `set -e`, which will immediately exit for any non-zero exit codes - # That's generally correct, but `git diff --no-index` will return 1 to indicate differences between the files. That's correct - # and expected behavior. - git diff --no-index --dst-prefix="" "infra/$APP_NAME" template-infra/infra/app > "template-infra/$APP_NAME.patch" || true - - # This is the second part that allows comparing between directories with different names - # -p3 strips the first 3 path fragments from filenames - # Ex: if the filename is `a/infra/app/app-config/dev.tf`, then -p3 causes it to become: `app-config/dev.tf` - # --directory="infra/$APP_NAME" prepends path parts to the filename - # Ex: if the filename is `a/infra/app/app-config/dev.tf`, then --directory causes it to become: `/infra/$APP_NAME/app-config/dev.tf` - # This is the stat version of the command to output the changes - STAT_COMMAND="git --no-pager apply --stat -p3 --directory=infra/$APP_NAME --allow-empty template-infra/$APP_NAME.patch --exclude='*.tfbackend' --exclude='*.terraform*'" + # If the APP_NAME is not named `app`, then install a new app in template-infra to diff against + if [ "$APP_NAME" != "app" ]; then + curl "https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/install-app.sh" | bash -s -- $APP_NAME "template-infra/" + echo + fi + + # To create a git patch comparing a project's application: + # --no-index allows us comparison between differently-named or -nested directories + # --dst-prefix="" removes the destination prefix, essentially having the effect of + # removing the additional `template-infra/` path prefix, making the patch appliable + # || true is necessary because this bash script includes `set -e` option, which will + # immediately exit for any non-zero exit codes. That's generally correct, but + # `git diff --no-index` will return 1 to indicate differences between the files. + git diff --no-index --dst-prefix="" "infra/$APP_NAME" "template-infra/infra/$APP_NAME" > "template-infra/$APP_NAME.patch" || true + + # The stat version of the `git-apply` command`, used to output the changes to STDOUT + STAT_COMMAND="git --no-pager apply --stat --allow-empty template-infra/$APP_NAME.patch --exclude='*.tfbackend' --exclude='*.terraform*'" eval "$STAT_COMMAND" - # Actually running the command `git apply` - git apply -p3 --directory="infra/$APP_NAME" --allow-empty "template-infra/$APP_NAME.patch" --exclude="*.tfbackend" --exclude="*.terraform*" + # Actually run the `git apply` command + git apply --allow-empty "template-infra/$APP_NAME.patch" --exclude="*.tfbackend" --exclude="*.terraform*" # Increment step counter STEP_COUNT=$((STEP_COUNT+1)) - - # @TODO patch for CI/CD done +echo +echo "---------------------------------------------------------------------" +echo "$STEP_COUNT. Patching CI/CD" +echo "---------------------------------------------------------------------" +# This follows the same pattern as above +git diff --no-index --dst-prefix="" .github/workflows template-infra/.github/workflows > "template-infra/ci-$APP_NAME.patch" || true +STAT_COMMAND="git --no-pager apply --stat --allow-empty template-infra/ci-$APP_NAME.patch --exclude='.github/workflows/template-only*'" +eval "$STAT_COMMAND" +git apply --allow-empty "template-infra/ci-$APP_NAME.patch" --exclude=".github/workflows/template-only*" +STEP_COUNT=$((STEP_COUNT+1)) + echo echo "---------------------------------------------------------------------" echo "$STEP_COUNT. Cleaning up" @@ -202,10 +218,11 @@ rm main-template.patch echo "Removing git remote..." git remote rm upstream-template-infra -# echo "Cleaning up template-infra folder..." -# rm -rf template-infra +echo "Cleaning up template-infra folder..." +rm -rf template-infra echo echo "=====================================================================" echo "Done." echo "=====================================================================" +echo "Review all changes carefully using 'git diff' before committing" \ No newline at end of file From f8f8c28a48124ba2e93b65ead48905e82c84840d Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 09:14:52 -0700 Subject: [PATCH 52/89] Remove feature flag infra changes --- docs/infra/set-up-app-config.md | 2 +- docs/infra/set-up-app-service.md | 10 +++++----- infra/app/app-config/feature-flags.tf | 7 +------ 3 files changed, 7 insertions(+), 12 deletions(-) diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md index 851b9fca2..a6621a20d 100644 --- a/docs/infra/set-up-app-config.md +++ b/docs/infra/set-up-app-config.md @@ -25,7 +25,7 @@ Modify the following values in the application's `app-config/main.tf` (e.g. in ` * Set `has_incident_management_service` to `true` or `false` to indicate whether the application should integrate with an incident management service. By default, this is set to `false`. * Set the `account_names_by_environment` hash to map environments to AWS accounts. See [set up AWS accounts](./set-up-aws-accounts.md) for more information. -Optionally, to use [feature flags](/docs/feature-flags.md), modify the values in the application's `app-config/feature-flags.tf` (e.g. in `/infra//app-config/feature-flags.tf`). +To use [feature flags](/docs/feature-flags.md), modify the values in the application's `app-config/feature-flags.tf` (e.g. in `/infra//app-config/feature-flags.tf`). ### 3. Configure each environment diff --git a/docs/infra/set-up-app-service.md b/docs/infra/set-up-app-service.md index eb2458f38..c285c7604 100644 --- a/docs/infra/set-up-app-service.md +++ b/docs/infra/set-up-app-service.md @@ -4,11 +4,11 @@ Follow these instructions for **each application** (you can have one or more in The application service setup process will: -1. Configure an ECS Fargate Service and Task to host the application -2. Creates a load balancer for the application -2. Create an S3 bucket for general object storage -3. Set up CloudWatch for logging, monitoring, and alerts -3. Optionally, configure CloudWatch Evidently to support [feature flags](/docs/feature-flags.md) +* Configure an ECS Fargate Service and Task to host the application +* Creates a load balancer for the application +* Create an S3 bucket for general object storage +* Set up CloudWatch for logging, monitoring, and alerts +* Configure CloudWatch Evidently to support [feature flags](/docs/feature-flags.md) ## Prerequisites diff --git a/infra/app/app-config/feature-flags.tf b/infra/app/app-config/feature-flags.tf index a92d9f9c1..6224b2538 100644 --- a/infra/app/app-config/feature-flags.tf +++ b/infra/app/app-config/feature-flags.tf @@ -1,8 +1,3 @@ locals { - # Add feature flags as strings to the array. - # For instance: - # feature_flags = ["foo", "bar"] - # See the /docs/feature-flags.md for more information. - feature_flags = [] - + feature_flags = ["foo", "bar"] } From b2a7d06d6376ebc15d52b431198b74d9d08491aa Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 09:20:13 -0700 Subject: [PATCH 53/89] Remove network infra changes --- docs/infra/set-up-network-custom-domains.md | 5 +- infra/project-config/networks.tf | 65 +++++++++------------ 2 files changed, 30 insertions(+), 40 deletions(-) diff --git a/docs/infra/set-up-network-custom-domains.md b/docs/infra/set-up-network-custom-domains.md index 20fb938fb..203c25131 100644 --- a/docs/infra/set-up-network-custom-domains.md +++ b/docs/infra/set-up-network-custom-domains.md @@ -39,7 +39,10 @@ The custom domain configuration is defined as a `domain_config` object in [`/inf For example, a hosted zone of `platform-test.navateam.com` includes `platform-test.navateam.com`, `cdn.platform-test.navateam.com`, `notifications.platform-test.navateam.com`, `foo.bar.platform-test.navateam.com`, etc. -**For each network** you want to use a custom domain, set the `hosted_zone` value in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) to match the custom domain (or a subdomain of the custom domain) that you registered. +**For each network** you want to use a custom domain, in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf): + +* set the `hosted_zone` to match the custom domain (or a subdomain of the custom domain) that you registered +* set `manage_dns` to `true` ### 3. Update the network layer to create the hosted zones diff --git a/infra/project-config/networks.tf b/infra/project-config/networks.tf index 11795b22e..69051ead8 100644 --- a/infra/project-config/networks.tf +++ b/infra/project-config/networks.tf @@ -3,51 +3,37 @@ locals { dev = { database_subnet_group_name = "dev" - - # To configure custom domains, configure the following: - # - Set `manage_dns` to `true` - # - Set the `hosted_zone` value. - # A hosted zone represents a domain and all of its subdomains. For example, a - # hosted zone of foo.domain.com includes foo.domain.com, bar.foo.domain.com, etc. domain_config = { - # Set to `true` to - manage_dns = false - hosted_zone = "" - - # To configure HTTPS support, custom domains must be configured. - certificate_configs = {} + manage_dns = true + # Placeholder value for the hosted zone + # A hosted zone represents a domain and all of its subdomains. For example, a + # hosted zone of foo.domain.com includes foo.domain.com, bar.foo.domain.com, etc. + hosted_zone = "hosted.zone.for.dev.network.com" + + certificate_configs = { + # Example certificate configuration for a certificate that is managed by the project + # "sub.domain.com" = { + # source = "issued" + # } + + # Example certificate configuration for a certificate that is issued elsewhere and imported into the project + # (currently not supported, will be supported via https://github.com/navapbc/template-infra/issues/559) + # "platform-test-dev.navateam.com" = { + # source = "imported" + # private_key_ssm_name = "/certificates/sub.domain.com/private-key" + # certificate_body_ssm_name = "/certificates/sub.domain.com/certificate-body" + # } + } } - - # domain_config = { - # manage_dns = true - # # Placeholder value for the hosted zone - # # A hosted zone represents a domain and all of its subdomains. For example, a - # # hosted zone of foo.domain.com includes foo.domain.com, bar.foo.domain.com, etc. - # hosted_zone = "hosted.zone.for.dev.network.com" - - # certificate_configs = { - # # Example certificate configuration for a certificate that is managed by the project - # # "sub.domain.com" = { - # # source = "issued" - # # } - - # # Example certificate configuration for a certificate that is issued elsewhere and imported into the project - # # (currently not supported, will be supported via https://github.com/navapbc/template-infra/issues/559) - # # "platform-test-dev.navateam.com" = { - # # source = "imported" - # # private_key_ssm_name = "/certificates/sub.domain.com/private-key" - # # certificate_body_ssm_name = "/certificates/sub.domain.com/certificate-body" - # # } - # } - # } } staging = { database_subnet_group_name = "staging" domain_config = { - manage_dns = false - hosted_zone = "" + manage_dns = true + hosted_zone = "hosted.zone.for.staging.network.com" + certificate_configs = {} } } @@ -56,8 +42,9 @@ locals { database_subnet_group_name = "prod" domain_config = { - manage_dns = false - hosted_zone = "" + manage_dns = true + hosted_zone = "hosted.zone.for.prod.network.com" + certificate_configs = {} } } From ec0d6991860183b54cc76038a7aa44a67ed9a676 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 10:10:02 -0700 Subject: [PATCH 54/89] Improve technical writing --- README.md | 56 ++++++++++++++++++++++++------------------- docs/feature-flags.md | 4 ++-- 2 files changed, 33 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index fcec2ac99..45d3d4c76 100644 --- a/README.md +++ b/README.md @@ -2,77 +2,83 @@ ## Overview -This is a template repository to set up foundational infrastructure for your application in AWS. It is part of a collection of interoperable [Platform templates](https://github.com/navapbc/platform). +This template sets up foundational infrastructure for applications hosted on Amazon Web Services (AWS). It belongs to a collection of interoperable [Platform templates](https://github.com/navapbc/platform). -This template includes setup for: +Using this template will set up: -* **Team workflows** - templates for pull requests (PRs), architecture decision records (ADRs), and Makefiles. -* **Account level foundational infrastructure** - infrastructure for terraform backends, including an S3 bucket and DynamoDB table for storing and managing terraform state files. -* **Application infrastructure** - the infrastructure you need to set up a basic web app, such as a image container repository, load balancer, web service, and database. -* **CI for infra** - GitHub action that performs infra code checks, including linting, validation, and security compliance checks. -* **CD / Deployments** - infrastructure for continuous deployment, including: AWS account access for Github actions, scripts for building and publishing release artifacts, and a Github action for automated deployments from the main branch. -* **Documentation** - technical documentation for the decisions that went into all the defaults that come with the template. +* **Team workflows** - templates for pull requests (PRs), architecture decision records (ADRs), and Makefiles +* **Account level foundational infrastructure** - infrastructure for terraform backends, including an S3 bucket and DynamoDB table for storing and managing terraform state files +* **Application infrastructure** - the infrastructure you need to set up a basic web app, including container image repository, load balancer, web service, and database +* **Continuous integration (CI) for infrastructure** - GitHub action that performs infra code checks, including linting, validation, and security compliance checks +* **Continous deployment (CD)** - infrastructure for continuous deployment, including AWS account access for Github actions, scripts for building and publishing release artifacts, and a Github action for automated deployments from the main branch +* **Documentation** - technical documentation for the decisions that went into all the defaults that come with the template -By default, the system architecture will look like this (see [system architecture documentation](/docs/system-architecture.md) for more information): +By default, the system architecture looks like this (for more information, see [system architecture documentation](/docs/system-architecture.md)): ![System architecture](https://lucid.app/publicSegments/view/e5a36152-200d-4d95-888e-4cdbdab80d1b/image.png) ## Application Requirements -This template assumes that you have an application to deploy. See [application requirements](https://github.com/navapbc/template-infra/tree/main/template-only-docs/application-requirements.md) for more information on what is needed to use the infrastructure template. If you're using one of the [Platform application templates](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), these requirements are already met. +To be used with this template, applications must meet [these requirements](/template-only-docs/application-requirements.md). If you're using one of the [Platform application templates](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), these requirements are already met. ### Multiple Applications -This infrastructure supports deployment and CI/CD for projects with multiple applications. By default, the infrastructure assumes the project only has one application, named `app`. However, it's straightforward to include additional applications. See [how to add multiple applications](https://github.com/navapbc/template-infra/tree/main/template-only-docs/multiple-applications.md). +You can use this template with multiple applications. By default, the infrastructure assumes the project only has one application, named `app`. However, it's straightforward to [add additional applications](https://github.com/navapbc/template-infra/tree/main/template-only-docs/multiple-applications.md). ## Installation -To get started using the infrastructure template on your project, run the following command in your project's directory to execute the [download and install script](https://github.com/navapbc/template-infra/tree/main/template-only-bin/download-and-install-template.sh), which clones the template repository, copies the template files to your repository, and removes any files that are only relevant to the template itself: +This template assumes that you already have an application to deploy. + +To install this template to your project, run the following command in your project's directory: ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/download-and-install-template.sh | bash -s ``` +The [download and install script](https://github.com/navapbc/template-infra/tree/main/template-only-bin/download-and-install-template.sh) clones this template repository, copies the template files to your repository, and removes files that are only relevant to the template itself. + Now you're ready to set up the various pieces of your infrastructure. ## Setup -After downloading and installing the template into your project: +After downloading and installing this template into your project, take the following steps to complete setup: -1. Follow the steps in [infra/README.md](/infra/README.md) to setup the infrastructure for your application. -2. After setting up AWS resources, you can [set up GitHub Actions workflows for continuous integration](./template-only-docs/set-up-ci.md). -3. After configuring GitHub Actions, you can [set up continuous deployment](./template-only-docs/set-up-cd.md). +1. Follow the "First time initialization" steps in [infra/README.md](/infra/README.md) +2. [Set up GitHub Actions workflows for continuous integration](./template-only-docs/set-up-ci.md). +3. [Set up continuous deployment](./template-only-docs/set-up-cd.md). 4. At any point, [set up your team workflow](./template-only-docs/set-up-team-workflow.md). ## Updates -To apply template updates to your project, run +This template includes a bash script to update your project to a newer version of the template. The [update script](/template-only-bin/update-template.sh) assumes that your project is version-controlled using `git`. The script will edit your project files, but it will not run `git commit`. Please review all changes carefully. + +To update your project to a newer version of this template, run the following command: ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- ``` - is a required argument. It must be a comma-separated list (no spaces) of the apps in `/infra`. App names are expected to be hyphen-separated (i.e. kebab-case). - Examples: `app`, `app,app2`, `my-app,your-app` +`` is a required argument. It must be a comma-separated list (no spaces) of the apps in `/infra`. App names must be hyphen-separated (i.e. kebab-case). Examples: `app`, `app,app2`, `my-app,your-app`. -By default, the update script will update to the latest commit on the `main` branch in the template repo. If you want to update to a different branch, a specific commit, or a specific tag (e.g. a release tag), run this instead +By default, the update script will apply changes up to the latest commit to the `main` branch of this template repo. If you want to update to a different branch, a specific commit, or a specific tag (e.g. a release tag), run this instead: ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- ``` - should be the version of the template to install. This can be a branch, commit hash, or tag. - should be the type of provided. Defaults to `branch`. This can be: `branch`, `commit`, or `tag`. +`` should be the version of the template to install. This can be a branch, commit hash, or tag. +`` should be the type of `` provided. Defaults to `branch`. This can be: `branch`, `commit`, or `tag`. Examples: -- To update a project with one application named `app` to `main` in the template repo: + +* To update a project with one application named `app` to the latest commit to the `main` of this template repo: ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- app ``` -- To update a project with two applications to a specific commit: +* To update a project with two applications to a specific commit: ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- app,app2 d42963d007e55cc37ef666019428b1d06a25cf71 commit ``` -- To update a project with three applications to a tag: +* To update a project with three applications to a tag: ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- alpha,beta,gamma-three v0.8.0 tag ``` diff --git a/docs/feature-flags.md b/docs/feature-flags.md index 82a9549b8..4c053b742 100644 --- a/docs/feature-flags.md +++ b/docs/feature-flags.md @@ -8,7 +8,7 @@ This project leverages [Amazon CloudWatch Evidently](https://docs.aws.amazon.com ## Creating feature flags -The list of feature flags for an application is defined in the `feature_flags` property in its app-config module (in `/infra//app-config/feature-flags.tf`). To create a new feature flag, add a new string to that list. To remove a feature flag, remove the feature flag from the list. The set of feature flags will be updated on the next terraform apply of the service layer, or during the next deploy of the application. +The list of feature flags for an application is defined in the `feature_flags` property in its app-config module (in `/infra/[app_name]/app-config/feature-flags.tf`). To create a new feature flag, add a new string to that list. To remove a feature flag, remove the feature flag from the list. The set of feature flags will be updated on the next `terraform apply` of the service layer, or during the next deploy of the application. ## Querying feature flags in the application @@ -16,7 +16,7 @@ To determine whether a particular feature should be enabled or disabled for a gi ## Managing feature releases and partial rollouts via AWS Console -The system is designed to allow the managing of feature releases and partial rollouts outside of terraform, which empowers business owners and product managers to control enable and disable feature flags and adjust feature launch traffic percentages without needing to depend on the development team. +The system is designed to allow the managing of feature releases and partial rollouts outside of Terraform, which empowers business owners and product managers to control enable and disable feature flags and adjust feature launch traffic percentages without needing to depend on the development team. ### To enable or disable a feature From f74f890bb253c5bb3aa9d7b27f86cf5c109b4183 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 10:20:23 -0700 Subject: [PATCH 55/89] Improve technical writing --- README.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 45d3d4c76..19ce5416d 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ This template sets up foundational infrastructure for applications hosted on Amazon Web Services (AWS). It belongs to a collection of interoperable [Platform templates](https://github.com/navapbc/platform). -Using this template will set up: +This template includes: * **Team workflows** - templates for pull requests (PRs), architecture decision records (ADRs), and Makefiles * **Account level foundational infrastructure** - infrastructure for terraform backends, including an S3 bucket and DynamoDB table for storing and managing terraform state files @@ -18,11 +18,11 @@ By default, the system architecture looks like this (for more information, see [ ## Application Requirements -To be used with this template, applications must meet [these requirements](/template-only-docs/application-requirements.md). If you're using one of the [Platform application templates](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), these requirements are already met. +Applications must meet [these requirements](/template-only-docs/application-requirements.md) to be used with this template. If you're using one of the [Platform application templates](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), these requirements are already met. ### Multiple Applications -You can use this template with multiple applications. By default, the infrastructure assumes the project only has one application, named `app`. However, it's straightforward to [add additional applications](https://github.com/navapbc/template-infra/tree/main/template-only-docs/multiple-applications.md). +You can use this template with multiple applications. By default, this template assumes your project has one application named `app`. However, it's straightforward to [add additional applications](https://github.com/navapbc/template-infra/tree/main/template-only-docs/multiple-applications.md). ## Installation @@ -42,14 +42,14 @@ Now you're ready to set up the various pieces of your infrastructure. After downloading and installing this template into your project, take the following steps to complete setup: -1. Follow the "First time initialization" steps in [infra/README.md](/infra/README.md) -2. [Set up GitHub Actions workflows for continuous integration](./template-only-docs/set-up-ci.md). +1. Follow the "First time initialization" steps in [infra/README.md](/infra/README.md). +2. [Set up continuous integration](./template-only-docs/set-up-ci.md). 3. [Set up continuous deployment](./template-only-docs/set-up-cd.md). 4. At any point, [set up your team workflow](./template-only-docs/set-up-team-workflow.md). ## Updates -This template includes a bash script to update your project to a newer version of the template. The [update script](/template-only-bin/update-template.sh) assumes that your project is version-controlled using `git`. The script will edit your project files, but it will not run `git commit`. Please review all changes carefully. +This template includes a bash script to update your project to a newer version of the template. The [update script](/template-only-bin/update-template.sh) assumes that your project is version-controlled using `git`. The script will edit your project files, but it will not run `git commit`. After running the script, use `git diff` to review all changes carefully. To update your project to a newer version of this template, run the following command: @@ -59,7 +59,7 @@ curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only `` is a required argument. It must be a comma-separated list (no spaces) of the apps in `/infra`. App names must be hyphen-separated (i.e. kebab-case). Examples: `app`, `app,app2`, `my-app,your-app`. -By default, the update script will apply changes up to the latest commit to the `main` branch of this template repo. If you want to update to a different branch, a specific commit, or a specific tag (e.g. a release tag), run this instead: +By default, the update script will apply changes from the `main` branch of this template repo. If you want to update to a different branch, a specific commit, or a specific tag (e.g. a release tag), run this instead: ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- @@ -69,18 +69,18 @@ curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only Examples: -* To update a project with one application named `app` to the latest commit to the `main` of this template repo: +* If your project has one application named `app` and you want to update it to the `main` branch of this template repo, run: ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- app ``` -* To update a project with two applications to a specific commit: +* If your project has two applications named `app, app2` and you want to update to the commit `d42963d007e55cc37ef666019428b1d06a25cf71`, run: ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- app,app2 d42963d007e55cc37ef666019428b1d06a25cf71 commit ``` -* To update a project with three applications to a tag: +* If your project has three applications named `alpha,beta,gamma-three` and you want to update to the `v.0.8.0` release tag, run: ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- alpha,beta,gamma-three v0.8.0 tag ``` -**Remember:** Make sure to read the release notes in case there are breaking changes you need to address. +**Remember:** Read the release notes in case there are breaking changes you need to address. From 12ed36811ec714e851fd37d56469a6dbce865aad Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 10:26:52 -0700 Subject: [PATCH 56/89] Improve technical writing --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 19ce5416d..d0b2b282f 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ By default, the system architecture looks like this (for more information, see [ ## Application Requirements -Applications must meet [these requirements](/template-only-docs/application-requirements.md) to be used with this template. If you're using one of the [Platform application templates](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), these requirements are already met. +Applications must meet [these requirements](/template-only-docs/application-requirements.md) to be used with this template. If you're using a [Platform application template](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), these requirements are already met. ### Multiple Applications @@ -28,7 +28,7 @@ You can use this template with multiple applications. By default, this template This template assumes that you already have an application to deploy. -To install this template to your project, run the following command in your project's directory: +To install this template to your project, run the following command in your project's root directory: ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/download-and-install-template.sh | bash -s @@ -51,7 +51,7 @@ After downloading and installing this template into your project, take the follo This template includes a bash script to update your project to a newer version of the template. The [update script](/template-only-bin/update-template.sh) assumes that your project is version-controlled using `git`. The script will edit your project files, but it will not run `git commit`. After running the script, use `git diff` to review all changes carefully. -To update your project to a newer version of this template, run the following command: +To update your project to a newer version of this template, run the following command in your project's root directory: ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- @@ -78,9 +78,9 @@ Examples: curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- app,app2 d42963d007e55cc37ef666019428b1d06a25cf71 commit ``` -* If your project has three applications named `alpha,beta,gamma-three` and you want to update to the `v.0.8.0` release tag, run: +* If your project has three applications named `foo,bar,baz` and you want to update to the `v.0.8.0` release tag, run: ```bash - curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- alpha,beta,gamma-three v0.8.0 tag + curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- foo,bar,baz v0.8.0 tag ``` **Remember:** Read the release notes in case there are breaking changes you need to address. From ddf93241a62ec2eabd102ce30ad337b3ab8feac1 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 10:36:24 -0700 Subject: [PATCH 57/89] Improve technical writing --- infra/README.md | 50 ++++++++++++++++++++++++------------------------- 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/infra/README.md b/infra/README.md index 0dbaf97f1..0f27fd11a 100644 --- a/infra/README.md +++ b/infra/README.md @@ -1,10 +1,10 @@ # Overview -This project practices infrastructure-as-code and uses the [Terraform framework](https://www.terraform.io). This directory contains the infrastructure code for this project, including infrastructure for all application resources. This terraform project uses the [AWS provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs). It is based on the [Nava platform infrastructure template](https://github.com/navapbc/template-infra). +This project practices infrastructure-as-code and uses the [Terraform framework](https://www.terraform.io). This directory contains the infrastructure code for this project, including infrastructure for all application resources. This terraform project uses the [AWS provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs). ## 📂 Directory structure -The structure for the infrastructure code looks like this: +The directory structure looks like this: ```text infra/ Infrastructure code @@ -29,7 +29,7 @@ Details about terraform root modules and child modules are documented in [module ### 🧅 Infrastructure layers -The infrastructure template is designed to operate on different layers: +The infrastructure operates on different layers: - Account layer - Network layer @@ -45,13 +45,13 @@ This project has the following AWS environments: - `staging` - `prod` -The environments share the same root modules but will have different configurations. Backend configuration is saved as [`.tfbackend`](https://developer.hashicorp.com/terraform/language/settings/backends/configuration#file) files. Most `.tfbackend` files are named after the environment. For example, the `/service` infrastructure resources for the `dev` environment are configured via `dev.s3.tfbackend`. Resources for a module that are shared across environments, such as the build-repository, use `shared.s3.tfbackend`. Resources that are shared across the entire account (e.g. /infra/accounts) use `..s3.tfbackend`. +The environments share the same root modules but have different configurations. Backend configuration is saved as [`.tfbackend`](https://developer.hashicorp.com/terraform/language/settings/backends/configuration#file) files. Most `.tfbackend` files are named after the environment. For example, the `/service` infrastructure resources for the `dev` environment are configured via `dev.s3.tfbackend`. Resources for modules that are shared across environments (e.g. `/build-repository`), use `shared.s3.tfbackend`. Resources that are shared across an entire AWS account (e.g. `/infra/accounts`) use `..s3.tfbackend`. ### 🔀 Project workflow This project relies on Make targets in the [root Makefile](/Makefile), which in turn call shell scripts in [./bin](/bin). The shell scripts call terraform commands. Many of the shell scripts are also called by the [Github Actions CI/CD](/.github/workflows). -Generally you should use the Make targets or the underlying bin scripts, but you can call the underlying terraform commands if needed. See [making-infra-changes](/docs/infra/making-infra-changes.md) for more details. +Generally, you should use the Make targets or the underlying bin scripts, but, if needed, you can call the underlying terraform commands. For more details, see [making-infra-changes](/docs/infra/making-infra-changes.md). ## 💻 Development @@ -59,37 +59,37 @@ Generally you should use the Make targets or the underlying bin scripts, but you #### Prerequisites -* You'll need to have [installed this template](/README.md#installation) into an application that meets the [Application Requirements](/README.md#application-requirements) +* You'll need to have [installed this template](/README.md#installation) into an application that meets the [Application Requirements](/README.md#application-requirements). #### Instructions -To set up this project for the first time, if it has never been deployed to the target AWS account(s): +If this project has never been deployed to the target AWS account(s), complete the following steps: -1. [Configure the project](/infra/project-config/main.tf) (These values will be used in subsequent infra setup steps to namespace resources and add infrastructure tags) -2. [Set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md) -3. [Set up AWS account(s)](/docs/infra/set-up-aws-accounts.md) +1. [Configure the project](/infra/project-config/main.tf). +2. [Set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md). +3. [Set up AWS account(s)](/docs/infra/set-up-aws-accounts.md). 4. For each application: - 1. [Configure the application](/docs/infra/set-up-app-config.md) -5. [Set up the network(s)](/docs/infra/set-up-networks.md) + 1. [Configure the application](/docs/infra/set-up-app-config.md). +5. [Set up network(s)](/docs/infra/set-up-networks.md). 6. For each application: - 1. Optionally, [configure environment variables and secrets](/docs/infra/set-up-environment-variables-and-secrets.md) - 2. [Set up application build repository](/docs/infra/set-up-app-build-repository.md) + 1. [Configure environment variables and secrets](/docs/infra/set-up-environment-variables-and-secrets.md). + 2. [Set up application build repository](/docs/infra/set-up-app-build-repository.md). 3. For each environment: - 1. [Set up application database](/docs/infra/set-up-app-database.md) - 2. [Set up application service](/docs/infra/set-up-app-service.md) - 3. Optionally, [set up application monitoring alerts](/docs/infra/set-up-app-monitoring-alerts.md) - 4. Optionally, [set up application background jobs](/docs/infra/background-jobs.md) -7. Optionally, [set up custom domains](/docs/infra/set-up-network-custom-domains.md) -8. Optionally, [set up HTTPS support](/docs/infra/set-up-network-https.md) + 1. [Set up application database](/docs/infra/set-up-app-database.md). + 2. [Set up application service](/docs/infra/set-up-app-service.md). + 3. (Optional) [Set up application monitoring alerts](/docs/infra/set-up-app-monitoring-alerts.md). + 4. (Optional) [Set up application background jobs](/docs/infra/background-jobs.md). +7. (Optional) [Set up custom domains](/docs/infra/set-up-network-custom-domains.md). +8. (Optional) [Set up HTTPS support](/docs/infra/set-up-network-https.md). ### 🆕 New developer -To get set up as a new developer to the project, if it has already been deployed to the target AWS account(s): +If you are a new developer on this project and this project has already been deployed to the target AWS account(s), complete the following steps to set up your local development environment: -1. [Set up your infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md) -2. [Review how to make changes to infrastructure](/docs/infra/making-infra-changes.md) -3. Optionally, set up a [terraform workspace](/docs/infra/intro-to-terraform-workspaces.md) +1. [Set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md). +2. [Review how to make changes to infrastructure](/docs/infra/making-infra-changes.md). +3. (Optional) [Set up a terraform workspace](/docs/infra/intro-to-terraform-workspaces.md). ## 📇 Additional reading -Additional documentation can be found in the [documentation directory](/docs/infra). +Additional documentation is located in [documentation directory](/docs/infra). From b72f33f1f7c2059a2ae2e8f2e9b3fdafb2008c82 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 10:43:29 -0700 Subject: [PATCH 58/89] Improve technical writing --- README.md | 2 +- infra/README.md | 12 ++++++++---- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index d0b2b282f..c6f021670 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ After downloading and installing this template into your project, take the follo ## Updates -This template includes a bash script to update your project to a newer version of the template. The [update script](/template-only-bin/update-template.sh) assumes that your project is version-controlled using `git`. The script will edit your project files, but it will not run `git commit`. After running the script, use `git diff` to review all changes carefully. +This template includes a script to update your project to a newer version of the template. The [update script](/template-only-bin/update-template.sh) assumes that your project is version-controlled using `git`. The script will edit your project files, but it will not run `git commit`. After running the script, use `git diff` to review all changes carefully. To update your project to a newer version of this template, run the following command in your project's root directory: diff --git a/infra/README.md b/infra/README.md index 0f27fd11a..7b9e9447b 100644 --- a/infra/README.md +++ b/infra/README.md @@ -45,13 +45,17 @@ This project has the following AWS environments: - `staging` - `prod` -The environments share the same root modules but have different configurations. Backend configuration is saved as [`.tfbackend`](https://developer.hashicorp.com/terraform/language/settings/backends/configuration#file) files. Most `.tfbackend` files are named after the environment. For example, the `/service` infrastructure resources for the `dev` environment are configured via `dev.s3.tfbackend`. Resources for modules that are shared across environments (e.g. `/build-repository`), use `shared.s3.tfbackend`. Resources that are shared across an entire AWS account (e.g. `/infra/accounts`) use `..s3.tfbackend`. +The environments share the same root modules but have different configurations. Backend configuration is saved as [`.tfbackend`](https://developer.hashicorp.com/terraform/language/settings/backends/configuration#file) files. `.tfbackend` files are named as follows: + +* Most `.tfbackend` files are named after the environment. For example, resources in the `dev` environment for the `/service` module use `dev.s3.tfbackend`. +* Resources for modules that are shared across environments (e.g. `/build-repository`) use `shared.s3.tfbackend`. +* Resources that are shared across an entire AWS account (e.g. `/infra/accounts`) use `..s3.tfbackend`. ### 🔀 Project workflow -This project relies on Make targets in the [root Makefile](/Makefile), which in turn call shell scripts in [./bin](/bin). The shell scripts call terraform commands. Many of the shell scripts are also called by the [Github Actions CI/CD](/.github/workflows). +This project relies on Make targets in the [root Makefile](/Makefile), which in turn call scripts in [/bin](/bin). The scripts call terraform commands. Many of the scripts are also called by the [Github Actions CI/CD](/.github/workflows). -Generally, you should use the Make targets or the underlying bin scripts, but, if needed, you can call the underlying terraform commands. For more details, see [making-infra-changes](/docs/infra/making-infra-changes.md). +Generally, you should use the Make targets or the underlying scripts, but, if needed, you can call the underlying terraform commands. For more details, see [making infrastructure changes](/docs/infra/making-infra-changes.md). ## 💻 Development @@ -59,7 +63,7 @@ Generally, you should use the Make targets or the underlying bin scripts, but, i #### Prerequisites -* You'll need to have [installed this template](/README.md#installation) into an application that meets the [Application Requirements](/README.md#application-requirements). +* You'll need to have [installed](/README.md#installation) this template into an application that meets the [Application Requirements](/README.md#application-requirements). #### Instructions From 70afcb3d7c5067a49cf16e0a7718839ef842cb4c Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 12:24:00 -0700 Subject: [PATCH 59/89] Improve technical writing --- infra/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/infra/README.md b/infra/README.md index 7b9e9447b..41a812177 100644 --- a/infra/README.md +++ b/infra/README.md @@ -63,7 +63,7 @@ Generally, you should use the Make targets or the underlying scripts, but, if ne #### Prerequisites -* You'll need to have [installed](/README.md#installation) this template into an application that meets the [Application Requirements](/README.md#application-requirements). +* You have [installed](/README.md#installation) this template into an application that meets the [Application Requirements](/README.md#application-requirements). #### Instructions From 84662b9ae9c75116c05b39b867e9327644540777 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 15:03:38 -0700 Subject: [PATCH 60/89] Improve technical writing --- docs/infra/set-up-infrastructure-tools.md | 100 +++++++++++----------- infra/README.md | 6 +- 2 files changed, 51 insertions(+), 55 deletions(-) diff --git a/docs/infra/set-up-infrastructure-tools.md b/docs/infra/set-up-infrastructure-tools.md index c7e12fca5..a5a5bf275 100644 --- a/docs/infra/set-up-infrastructure-tools.md +++ b/docs/infra/set-up-infrastructure-tools.md @@ -1,42 +1,49 @@ # Set up infrastructure developer tools -If you are working on the infrastructure, you will need to complete these setup steps. +To work on the infrastructure, complete these steps. -## Prerequisites +## Instructions ### Install Terraform -[Terraform](https://www.terraform.io/) is an infrastructure as code (IaC) tool that allows you to build, change, and version infrastructure safely and efficiently. This includes both low-level components like compute instances, storage, and networking, as well as high-level components like DNS entries and SaaS features. +[Terraform](https://www.terraform.io/) is an infrastructure as code (IaC) tool that allows you to build, change, and version infrastructure safely and efficiently. This includes both low-level components, like compute instances, storage, and networking, as well as high-level components, like DNS entries and Software-as-a-Service (SaaS) features. -You may need different versions of Terraform since different projects may require different versions of Terraform. The best way to manage Terraform versions is with [Terraform Version Manager (tfenv)](https://github.com/tfutils/tfenv). +You may need to install different versions of Terraform on your machine because different projects may require different versions of Terraform. The best way to manage Terraform versions is with [Terraform Version Manager (tfenv)](https://github.com/tfutils/tfenv). -To install via [Homebrew](https://brew.sh/) +Follow these steps to set up tfenv: -```bash -brew install tfenv -``` - -Then install the version of Terraform you need. - -```bash -tfenv install 1.4.6 -``` +1. Use [Homebrew](https://brew.sh/) to install tfenv: + ```bash + brew install tfenv + ``` +2. Install the version of Terraform you need: + ```bash + tfenv install 1.4.6 + ``` If you are unfamiliar with Terraform, check out this [basic introduction to Terraform](./intro-to-terraform.md). ### Install AWS CLI -The [AWS Command Line Interface (AWS CLI)](https://aws.amazon.com/cli/) is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts. Install the AWS commmand line tool by following the instructions found here: +The [AWS Command Line Interface (AWS CLI)](https://aws.amazon.com/cli/) is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts. -- [Install AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) +Install the AWS CLI by following the [AWS installation instructions](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). ### Install Go The [Go programming language](https://go.dev/dl/) is required to run [Terratest](https://terratest.gruntwork.io/), the unit test framework for Terraform. +Use Homebrew to install go: + +```bash +brew install golang +``` + ### Install GitHub CLI -The [GitHub CLI](https://cli.github.com/) is useful for automating certain operations for GitHub such as with GitHub actions. This is needed to run [check-github-actions-auth.sh](/bin/check-github-actions-auth.sh) +The [GitHub CLI](https://cli.github.com/) is useful for automating certain operations on GitHub, such as GitHub actions. For example, you need the Github CLI to run [check-github-actions-auth.sh](/bin/check-github-actions-auth.sh). + +Use Homebrew to install the GitHub CLI: ```bash brew install gh @@ -44,52 +51,41 @@ brew install gh ### Install linters -We have several optional utilities for running infrastructure linters locally. These are run as part of the CI pipeline, therefore, it is often simpler to test them locally first. +The following linters are run as part of the CI pipeline: * [Shellcheck](https://github.com/koalaman/shellcheck) * [actionlint](https://github.com/rhysd/actionlint) * [markdown-link-check](https://github.com/tcort/markdown-link-check) +To install and run them locally, run: + ```bash brew install shellcheck brew install actionlint make infra-lint ``` -## AWS Authentication - -In order for Terraform to authenticate with your accounts you will need to configure your aws credentials using the AWS CLI or manually create your config and credentials file. If you need to manage multiple credentials or create named profiles for use with different environments you can add the `--profile` option. - -There are multiple ways to authenticate, but we recommend creating a separate profile for your project in your AWS credentials file, and setting your local environment variable `AWS_PROFILE` to the profile name. We recommend using [direnv](https://direnv.net/) to manage local environment variables. -**Credentials should be located in ~/.aws/credentials** (Linux & Mac) or **%USERPROFILE%\.aws\credentials** (Windows) - -### Examples - -```bash -$ aws configure -AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE -AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY -Default region name [None]: us-east-2 -Default output format [None]: json -``` - -**Using the above command will create a [default] profile.** - -```bash -$ aws configure --profile dev -AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE -AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY -Default region name [None]: us-east-2 -Default output format [None]: json -``` - -**Using the above command will create a [dev] profile.** - -Once you're done, verify access by running the following command to print out information about the AWS IAM user you authenticated as. - -```bash -aws sts get-caller-identity -``` +### Authenticate with AWS + +To use terraform with your AWS accounts, you must configure your AWS credentials. There are multiple ways to authenticate with AWS, but we recommend the following process: + +1. Use the AWS CLI command `aws configure --profile ` to create a separate profile for each AWS account. `aws configure` will store your credentials in ~/.aws/credentials** (Linux & Mac) or **%USERPROFILE%\.aws\credentials** (Windows). For example, to create a profile named `my-aws-account`, run: + ```bash + $ aws configure --profile my-aws-account + AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE + AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY + Default region name [None]: us-east-2 + Default output format [None]: json + ``` +2. Set the local environment variable `AWS_PROFILE` to the profile name. For example, to set the `AWS_PROFILE` environment variable to `my-aws-account`, run: + ```bash + export AWS_PROFILE=my-aws-account + ``` +3. (Optional) Use the [direnv](https://direnv.net/) to manage local environment variables. Instead of directly exporting environment variables on your machine, allow direnv to automatically set environment variables depending on the directory you are working in. +4. Verify access by running the following command. It should print out the profile name you set in Step 1. + ```bash + aws sts get-caller-identity + ``` ### References diff --git a/infra/README.md b/infra/README.md index 41a812177..175fe32bd 100644 --- a/infra/README.md +++ b/infra/README.md @@ -1,6 +1,6 @@ # Overview -This project practices infrastructure-as-code and uses the [Terraform framework](https://www.terraform.io). This directory contains the infrastructure code for this project, including infrastructure for all application resources. This terraform project uses the [AWS provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs). +This project practices infrastructure-as-code (IaC) and uses the [Terraform framework](https://www.terraform.io). This directory contains the infrastructure code for this project, including infrastructure for all application resources. This terraform project uses the [AWS provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs). ## 📂 Directory structure @@ -8,7 +8,7 @@ The directory structure looks like this: ```text infra/ Infrastructure code - accounts/ [Root module] IaC and IAM resources + accounts/ [Root module] IaC and Identity Access Management (IAM) resources [APP_NAME]/ Application directory: infrastructure for the [APP_NAME] application modules/ Reusable child modules networks/ [Root module] Account level network config (shared across all apps, environments, and terraform workspaces) @@ -18,7 +18,7 @@ Each application directory contains the following: ```text app-config/ Application-level configuration for the application resources (different config for different environments) - build-repository/ [Root module] Docker image repository for the application (shared across environments and terraform workspaces) + build-repository/ [Root module] Container image repository for the application (shared across environments and terraform workspaces) database/ [Root module] Configuration for database (different config for different environments) service/ [Root module] Configuration for containers, such as load balancer, application service (different config for different environments) ``` From d0463bc77c4e15833647d73574b19820ed250527 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 15:07:59 -0700 Subject: [PATCH 61/89] Improve technical writing --- docs/infra/set-up-infrastructure-tools.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/docs/infra/set-up-infrastructure-tools.md b/docs/infra/set-up-infrastructure-tools.md index a5a5bf275..5e4d44521 100644 --- a/docs/infra/set-up-infrastructure-tools.md +++ b/docs/infra/set-up-infrastructure-tools.md @@ -1,6 +1,6 @@ # Set up infrastructure developer tools -To work on the infrastructure, complete these steps. +Complete these steps to work on the infrastructure. ## Instructions @@ -8,9 +8,7 @@ To work on the infrastructure, complete these steps. [Terraform](https://www.terraform.io/) is an infrastructure as code (IaC) tool that allows you to build, change, and version infrastructure safely and efficiently. This includes both low-level components, like compute instances, storage, and networking, as well as high-level components, like DNS entries and Software-as-a-Service (SaaS) features. -You may need to install different versions of Terraform on your machine because different projects may require different versions of Terraform. The best way to manage Terraform versions is with [Terraform Version Manager (tfenv)](https://github.com/tfutils/tfenv). - -Follow these steps to set up tfenv: +You may need to install different versions of terraform on your machine because different projects may require different versions. We recommend managing terraform with [Terraform Version Manager (tfenv)](https://github.com/tfutils/tfenv). 1. Use [Homebrew](https://brew.sh/) to install tfenv: ```bash @@ -21,7 +19,7 @@ Follow these steps to set up tfenv: tfenv install 1.4.6 ``` -If you are unfamiliar with Terraform, check out this [basic introduction to Terraform](./intro-to-terraform.md). +If you are unfamiliar with terraform, check out this [basic introduction](./intro-to-terraform.md). ### Install AWS CLI @@ -69,7 +67,7 @@ make infra-lint To use terraform with your AWS accounts, you must configure your AWS credentials. There are multiple ways to authenticate with AWS, but we recommend the following process: -1. Use the AWS CLI command `aws configure --profile ` to create a separate profile for each AWS account. `aws configure` will store your credentials in ~/.aws/credentials** (Linux & Mac) or **%USERPROFILE%\.aws\credentials** (Windows). For example, to create a profile named `my-aws-account`, run: +1. Use the AWS CLI command `aws configure --profile ` to create a separate profile for each AWS account. `aws configure` will store your credentials in `~/.aws/credentials` (Linux & Mac) or `%USERPROFILE%\.aws\credentials` (Windows). For example, to create a profile named `my-aws-account`, run: ```bash $ aws configure --profile my-aws-account AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE @@ -81,7 +79,7 @@ To use terraform with your AWS accounts, you must configure your AWS credentials ```bash export AWS_PROFILE=my-aws-account ``` -3. (Optional) Use the [direnv](https://direnv.net/) to manage local environment variables. Instead of directly exporting environment variables on your machine, allow direnv to automatically set environment variables depending on the directory you are working in. +3. (Optional) Use the [direnv](https://direnv.net/) tool to manage local environment variables. Instead of directly exporting environment variables on your machine, allow direnv to automatically set environment variables depending on the directory you are working in. 4. Verify access by running the following command. It should print out the profile name you set in Step 1. ```bash aws sts get-caller-identity From 24d12f6b866b88d67734d6d6f80234d582336d43 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 15:16:01 -0700 Subject: [PATCH 62/89] Improve technical writing --- docs/infra/set-up-aws-accounts.md | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/docs/infra/set-up-aws-accounts.md b/docs/infra/set-up-aws-accounts.md index 1394ec0fc..0d92ba79f 100644 --- a/docs/infra/set-up-aws-accounts.md +++ b/docs/infra/set-up-aws-accounts.md @@ -1,36 +1,34 @@ # Set up AWS accounts -The infrastructure supports managing resources across different [AWS accounts](https://docs.aws.amazon.com/accounts/latest/reference/accounts-welcome.html) by mapping different environments (e.g. `dev`, `staging`, `prod`) to specified AWS accounts. Resources that are shared across environments, such as build repositories, are also deployed into a specified AWS account. - -The environment-to-AWS-account mapping is specified for each application deployed by this infrastructure. This means that multiple applications can share AWS accounts or be deployed into different AWS accounts. - The AWS accounts setup process will: 1. Help you decide on your environments 2. Help you decide on your AWS account strategy -3. Configure all AWS accounts +3. Configure all of your AWS accounts ## Prerequisites -* You'll need to have [set up infrastructure tools](./set-up-infrastructure-tools.md), like Terraform, AWS CLI, and AWS authentication +* You have [set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md). ## Instructions ### 1. Decide on environments -Even though environment-to-AWS-account mapping is specified per-application, the names of the environments must be consistent across applications. Decide now what you would like the environments in your project to be. By default, the environments are: +The names of your environments must be consistent for all applications. By default, the environments are: -* dev -* staging -* prod +* `dev` +* `staging` +* `prod` -You can always add new environments or delete existing environments later. +You can changes these names, add additional environments, or delete any these environments. You must have at least one environment. You can always add new environments or delete existing environments later. ### 2. Decide on AWS account strategy -Some resources, such as a build repository, are shared across environments. These resources can be deployed to the same AWS account as any of the other environments or they can be deployed to a separate environment. In the following diagrams, these are represented by the box labeled `shared`. +This template supports managing resources in multiple [AWS accounts](https://docs.aws.amazon.com/accounts/latest/reference/accounts-welcome.html). To do so, each environment (e.g. `dev`, `staging`, `prod`) is mapped to a specified AWS account. Resources that are shared across environments, such as build repositories, are also explicitly mapped to a specified AWS account. In the following diagrams, these are represented by the box labeled `shared`. + +The environment-to-AWS-account mapping is specified for each application. Multiple applications can share AWS accounts. -A simple project might have only one AWS account and all environments should be deployed to this environment. This mapping might look like this: +A simple project might have only one AWS account and all environments should be deployed to this environment: ```mermaid flowchart TD @@ -40,7 +38,7 @@ flowchart TD prod --> my_aws_account ``` -A more complex project might have separate AWS accounts for environment, enhancing security by isolating each environment into completely separate AWS accounts. This mapping might look like this: +A more complex project might have separate AWS accounts for environment, enhancing security by isolating each environment into completely separate AWS accounts: ```mermaid flowchart TD @@ -64,7 +62,7 @@ Decide on the strategy that is appropriate for your project. ### 3. Ensure AWS account(s) have been created -For **each AWS account** you wish to use, ensure it has been created in AWS and you are able to authenticate to it. +For **each AWS account** you wish to use, ensure the AWS account has been created and you are able to authenticate to it. ### 4. Set up AWS account From 2bf687d11edd4eb370bf28dbb832c7726317ee3c Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 15:23:00 -0700 Subject: [PATCH 63/89] Consistently capitalize 'Terraform' --- README.md | 2 +- docs/infra/database-access-control.md | 2 +- docs/infra/intro-to-terraform.md | 2 +- docs/infra/making-infra-changes.md | 10 +++++----- docs/infra/module-architecture.md | 2 +- docs/infra/set-up-app-database.md | 2 +- docs/infra/set-up-app-service.md | 2 +- docs/infra/set-up-aws-account.md | 2 +- docs/infra/set-up-infrastructure-tools.md | 4 ++-- docs/infra/set-up-network.md | 2 +- docs/system-architecture.md | 2 +- infra/README.md | 14 +++++++------- 12 files changed, 23 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index c6f021670..fa8ead7f8 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ This template sets up foundational infrastructure for applications hosted on Ama This template includes: * **Team workflows** - templates for pull requests (PRs), architecture decision records (ADRs), and Makefiles -* **Account level foundational infrastructure** - infrastructure for terraform backends, including an S3 bucket and DynamoDB table for storing and managing terraform state files +* **Account level foundational infrastructure** - infrastructure for Terraform backends, including an S3 bucket and DynamoDB table for storing and managing Terraform state files * **Application infrastructure** - the infrastructure you need to set up a basic web app, including container image repository, load balancer, web service, and database * **Continuous integration (CI) for infrastructure** - GitHub action that performs infra code checks, including linting, validation, and security compliance checks * **Continous deployment (CD)** - infrastructure for continuous deployment, including AWS account access for Github actions, scripts for building and publishing release artifacts, and a Github action for automated deployments from the main branch diff --git a/docs/infra/database-access-control.md b/docs/infra/database-access-control.md index 5032d16c9..436caa22f 100644 --- a/docs/infra/database-access-control.md +++ b/docs/infra/database-access-control.md @@ -5,7 +5,7 @@ The master user password is managed by Amazon RDS and Secrets Manager. Managing RDS master user passwords with Secrets Manager provides the following security benefits: * RDS rotates database credentials regularly, without requiring application changes. -* Secrets Manager secures database credentials from human access and plain text view. The master password is not even in the terraform state file. +* Secrets Manager secures database credentials from human access and plain text view. The master password is not even in the Terraform state file. For more information about the benefits, see [Benefits of managing master user passwords with Secrets Manager](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-secrets-manager.html#rds-secrets-manager-benefits). diff --git a/docs/infra/intro-to-terraform.md b/docs/infra/intro-to-terraform.md index 6736248c5..f5165ffa3 100644 --- a/docs/infra/intro-to-terraform.md +++ b/docs/infra/intro-to-terraform.md @@ -16,7 +16,7 @@ The `terraform destroy` command is a convenient way to destroy all remote object ⚠️ WARNING! ⚠️ This is a destructive command! As a best practice, it's recommended that you comment out resources in non-development environments rather than running this command. `terraform destroy` should only be used as a way to cleanup a development environment. e.g. a developers workspace after they are done with it. -For more information about terraform commands follow the link below: +For more information about Terraform commands follow the link below: - [Basic CLI Features](https://www.terraform.io/cli/commands) diff --git a/docs/infra/making-infra-changes.md b/docs/infra/making-infra-changes.md index 9998569e3..1d417f3fb 100644 --- a/docs/infra/making-infra-changes.md +++ b/docs/infra/making-infra-changes.md @@ -2,7 +2,7 @@ ## Requirements -First read [Module Architecture](./module-architecture.md) to understand how the terraform modules are structured. +First read [Module Architecture](./module-architecture.md) to understand how the Terraform modules are structured. ## Using make targets (recommended) @@ -34,9 +34,9 @@ TF_CLI_ARGS_apply='-input=false -auto-approve' make infra-update-app-service APP TF_CLI_ARGS_apply='-var=image_tag=abcdef1' make infra-update-app-service APP_NAME=app ENVIRONMENT=dev ``` -## Using terraform CLI wrapper scripts +## Using Terraform CLI wrapper scripts -An alternative to using the Makefile is to directly use the terraform wrapper scripts that the Makefile uses: +An alternative to using the Makefile is to directly use the Terraform wrapper scripts that the Makefile uses: ```bash project-root$ ./bin/terraform-init.sh app/service dev @@ -46,9 +46,9 @@ project-root$ ./bin/terraform-init-and-apply.sh app/service dev # calls init an Look in the script files for more details on usage. -## Using terraform CLI directly +## Using Terraform CLI directly -Finally, if the wrapper scripts don't meet your needs, you can always run terraform directly from the root module directory. You may need to do this if you are running terraform commands other than `terraform plan` and `terraform apply`, such as `terraform import`, `terraform taint`, etc. To do this, you'll need to pass in the appropriate `tfvars` and `tfbackend` files to `terraform init` and `terraform apply`. For example, to make changes to the application's service resources in the dev environment, cd to the `infra/app/service` directory and run: +Finally, if the wrapper scripts don't meet your needs, you can always run Terraform directly from the root module directory. You may need to do this if you are running terraform commands other than `terraform plan` and `terraform apply`, such as `terraform import`, `terraform taint`, etc. To do this, you'll need to pass in the appropriate `tfvars` and `tfbackend` files to `terraform init` and `terraform apply`. For example, to make changes to the application's service resources in the dev environment, cd to the `infra/app/service` directory and run: ```bash infra/app/service$ terraform init -backend-config=dev.s3.tfbackend diff --git a/docs/infra/module-architecture.md b/docs/infra/module-architecture.md index b4b10a688..423fa0c9b 100644 --- a/docs/infra/module-architecture.md +++ b/docs/infra/module-architecture.md @@ -85,7 +85,7 @@ When deciding which layer to put an infrastructure resource in, follow the follo * **Consider policy constraints on what resources the project team is authorized to manage:** Different categories of resources may have different requirements on who is allowed to create and manage those resources. Resources that the project team are not allowed to manage directly should not be mixed with resources that the project team needs to manage directly. -* **Consider out of band dependencies:** Put infrastructure resources that require steps outside of terraform to be completed configured in layers that are upstream to resources that depend on those completed resources. For example, after creating a database cluster, the database schemas, roles, and privileges need to be configured before they can be used by a downstream service. Therefore database resources should be separate from the service layer so that the database can be configured fully before attempting to create the service layer resources. +* **Consider out of band dependencies:** Put infrastructure resources that require steps outside of Terraform to be completed configured in layers that are upstream to resources that depend on those completed resources. For example, after creating a database cluster, the database schemas, roles, and privileges need to be configured before they can be used by a downstream service. Therefore database resources should be separate from the service layer so that the database can be configured fully before attempting to create the service layer resources. ## Making changes to infrastructure diff --git a/docs/infra/set-up-app-database.md b/docs/infra/set-up-app-database.md index 4dd42fee1..37accf0a4 100644 --- a/docs/infra/set-up-app-database.md +++ b/docs/infra/set-up-app-database.md @@ -47,7 +47,7 @@ make infra-configure-app-database APP_NAME= ENVIRONMENT= ### 3. Create database resources -Now run the following commands to create the resources. Review the terraform before confirming "yes" to apply the changes. This can take over 5 minutes. +Now run the following commands to create the resources. Review the Terraform before confirming "yes" to apply the changes. This can take over 5 minutes. ```bash make infra-update-app-database APP_NAME= ENVIRONMENT= diff --git a/docs/infra/set-up-app-service.md b/docs/infra/set-up-app-service.md index c285c7604..e0dca86a4 100644 --- a/docs/infra/set-up-app-service.md +++ b/docs/infra/set-up-app-service.md @@ -64,7 +64,7 @@ Copy the image tag name that was published. You'll need this in the next step. ### 4. Create application resources with the image tag that was published -Now run the following commands to create the resources, using the image tag that was published from the previous step. Review the terraform before confirming "yes" to apply the changes. +Now run the following commands to create the resources, using the image tag that was published from the previous step. Review the Terraform before confirming "yes" to apply the changes. ```bash TF_CLI_ARGS_apply="-var=image_tag=" make infra-update-app-service APP_NAME= ENVIRONMENT= diff --git a/docs/infra/set-up-aws-account.md b/docs/infra/set-up-aws-account.md index f8eca00e6..53a1a57cb 100644 --- a/docs/infra/set-up-aws-account.md +++ b/docs/infra/set-up-aws-account.md @@ -42,7 +42,7 @@ This command will create the S3 tfstate bucket and the GitHub OIDC provider. It ## Making changes to the account -If you make changes to the account terraform and want to apply those changes, run +If you make changes to the account Terraform and want to apply those changes, run ```bash make infra-update-current-account diff --git a/docs/infra/set-up-infrastructure-tools.md b/docs/infra/set-up-infrastructure-tools.md index 5e4d44521..b939b4a39 100644 --- a/docs/infra/set-up-infrastructure-tools.md +++ b/docs/infra/set-up-infrastructure-tools.md @@ -19,7 +19,7 @@ You may need to install different versions of terraform on your machine because tfenv install 1.4.6 ``` -If you are unfamiliar with terraform, check out this [basic introduction](./intro-to-terraform.md). +If you are unfamiliar with Terraform, check out this [basic introduction](./intro-to-terraform.md). ### Install AWS CLI @@ -65,7 +65,7 @@ make infra-lint ### Authenticate with AWS -To use terraform with your AWS accounts, you must configure your AWS credentials. There are multiple ways to authenticate with AWS, but we recommend the following process: +To use Terraform with your AWS accounts, you must configure your AWS credentials. There are multiple ways to authenticate with AWS, but we recommend the following process: 1. Use the AWS CLI command `aws configure --profile ` to create a separate profile for each AWS account. `aws configure` will store your credentials in `~/.aws/credentials` (Linux & Mac) or `%USERPROFILE%\.aws\credentials` (Windows). For example, to create a profile named `my-aws-account`, run: ```bash diff --git a/docs/infra/set-up-network.md b/docs/infra/set-up-network.md index 8919c368c..f74da8c7d 100644 --- a/docs/infra/set-up-network.md +++ b/docs/infra/set-up-network.md @@ -40,7 +40,7 @@ make infra-configure-network NETWORK_NAME= ### 3. Create network resources -Now run the following commands to create the resources. Review the terraform before confirming "yes" to apply the changes. +Now run the following commands to create the resources. Review the Terraform before confirming "yes" to apply the changes. ```bash make infra-update-network NETWORK_NAME= diff --git a/docs/system-architecture.md b/docs/system-architecture.md index ba22b5f55..55400d4bb 100644 --- a/docs/system-architecture.md +++ b/docs/system-architecture.md @@ -15,7 +15,7 @@ This diagram shows the system architecture. [🔒 Make a copy of this Lucid temp * **GitHub** — Source code repository. Also responsible for Continuous Integration (CI) and Continuous Delivery (CD) workflows. GitHub Actions builds and deploys releases to an Amazon ECR registry that stores Docker container images for the application service. * **Incident Management Service** — Incident management service (e.g. PagerDuty or Splunk On-Call) for managing on-call schedules and paging engineers for urgent production issues. * **Service** — Amazon ECS service running the application. -* **Terraform Backend Bucket** — Amazon S3 bucket used to store terraform state files. +* **Terraform Backend Bucket** — Amazon S3 bucket used to store Terraform state files. * **Terraform Locks DynamoDB Table** — Amazon DynamoDB table used to manage concurrent access to terraform state files. * **VPC Endpoints** — VPC endpoints are used by the Database Role Manager to access Amazon Services without traffic leaving the VPC. * **VPC Network** — Amazon VPC network. diff --git a/infra/README.md b/infra/README.md index 175fe32bd..fe46b8736 100644 --- a/infra/README.md +++ b/infra/README.md @@ -1,6 +1,6 @@ # Overview -This project practices infrastructure-as-code (IaC) and uses the [Terraform framework](https://www.terraform.io). This directory contains the infrastructure code for this project, including infrastructure for all application resources. This terraform project uses the [AWS provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs). +This project practices infrastructure-as-code (IaC) and uses the [Terraform framework](https://www.terraform.io). This directory contains the infrastructure code for this project, including infrastructure for all application resources. This Terraform project uses the [AWS provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs). ## 📂 Directory structure @@ -11,19 +11,19 @@ infra/ Infrastructure code accounts/ [Root module] IaC and Identity Access Management (IAM) resources [APP_NAME]/ Application directory: infrastructure for the [APP_NAME] application modules/ Reusable child modules - networks/ [Root module] Account level network config (shared across all apps, environments, and terraform workspaces) + networks/ [Root module] Account level network config (shared across all apps, environments, and Terraform workspaces) ``` Each application directory contains the following: ```text app-config/ Application-level configuration for the application resources (different config for different environments) - build-repository/ [Root module] Container image repository for the application (shared across environments and terraform workspaces) + build-repository/ [Root module] Container image repository for the application (shared across environments and Terraform workspaces) database/ [Root module] Configuration for database (different config for different environments) service/ [Root module] Configuration for containers, such as load balancer, application service (different config for different environments) ``` -Details about terraform root modules and child modules are documented in [module-architecture](/docs/infra/module-architecture.md). +Details about Terraform root modules and child modules are documented in [module-architecture](/docs/infra/module-architecture.md). ## 🏗️ Project architecture @@ -53,9 +53,9 @@ The environments share the same root modules but have different configurations. ### 🔀 Project workflow -This project relies on Make targets in the [root Makefile](/Makefile), which in turn call scripts in [/bin](/bin). The scripts call terraform commands. Many of the scripts are also called by the [Github Actions CI/CD](/.github/workflows). +This project relies on Make targets in the [root Makefile](/Makefile), which in turn call scripts in [/bin](/bin). The scripts call Terraform commands. Many of the scripts are also called by the [Github Actions CI/CD](/.github/workflows). -Generally, you should use the Make targets or the underlying scripts, but, if needed, you can call the underlying terraform commands. For more details, see [making infrastructure changes](/docs/infra/making-infra-changes.md). +Generally, you should use the Make targets or the underlying scripts, but, if needed, you can call the underlying Terraform commands. For more details, see [making infrastructure changes](/docs/infra/making-infra-changes.md). ## 💻 Development @@ -92,7 +92,7 @@ If you are a new developer on this project and this project has already been dep 1. [Set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md). 2. [Review how to make changes to infrastructure](/docs/infra/making-infra-changes.md). -3. (Optional) [Set up a terraform workspace](/docs/infra/intro-to-terraform-workspaces.md). +3. (Optional) [Set up a Terraform workspace](/docs/infra/intro-to-terraform-workspaces.md). ## 📇 Additional reading From fc1d704aa74eb958b5015f3f2161ac3e0a006957 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 15:27:15 -0700 Subject: [PATCH 64/89] Make setup lists and prereq lists consistent --- docs/infra/set-up-app-database.md | 18 +++++++++--------- docs/infra/set-up-app-monitoring-alerts.md | 8 ++++---- docs/infra/set-up-app-service.md | 10 +++++----- docs/infra/set-up-aws-account.md | 12 ++++++------ docs/infra/set-up-aws-accounts.md | 6 +++--- docs/infra/set-up-network-custom-domains.md | 14 +++++++------- docs/infra/set-up-network-https.md | 6 +++--- docs/infra/set-up-network.md | 12 ++++++------ docs/infra/set-up-networks.md | 8 ++++---- 9 files changed, 47 insertions(+), 47 deletions(-) diff --git a/docs/infra/set-up-app-database.md b/docs/infra/set-up-app-database.md index 37accf0a4..434d3cc39 100644 --- a/docs/infra/set-up-app-database.md +++ b/docs/infra/set-up-app-database.md @@ -4,18 +4,18 @@ Follow these instructions for **each application** (you can have one or more in The database setup process will: -1. Configure and deploy an application database cluster using [Amazon Aurora Serverless V2](https://aws.amazon.com/rds/aurora/serverless/) -2. Create a [PostgreSQL schema](https://www.postgresql.org/docs/current/ddl-schemas.html) `app` to contain tables used by the application -3. Create an IAM policy that allows IAM roles with that policy attached to [connect to the database using IAM authentication](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.html) -4. Create an [AWS Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html), the "role manager", for provisioning the [PostgreSQL database users](https://www.postgresql.org/docs/8.0/user-manag.html) that will be used by the application service and by the migrations task -5. Invoke the role manager function to create the `app` and `migrator` Postgres users +* Configure and deploy an application database cluster using [Amazon Aurora Serverless V2](https://aws.amazon.com/rds/aurora/serverless/) +* Create a [PostgreSQL schema](https://www.postgresql.org/docs/current/ddl-schemas.html) `app` to contain tables used by the application +* Create an IAM policy that allows IAM roles with that policy attached to [connect to the database using IAM authentication](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.html) +* Create an [AWS Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html), the "role manager", for provisioning the [PostgreSQL database users](https://www.postgresql.org/docs/8.0/user-manag.html) that will be used by the application service and by the migrations task +* Invoke the role manager function to create the `app` and `migrator` Postgres users ## Prerequisites -* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) -* You'll need to have [configured the application](/infra/app/app-config/main.tf) -* You'll need to have [set up the network(s)](./set-up-networks.md) -* You'll need to have [pip](https://pypi.org/project/pip/) installed (pip is needed to download dependencies for the role manager Lambda function) +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You have [configured the application](/infra/app/app-config/main.tf). +* You have [set up the network(s)](./set-up-networks.md). +* You have [pip](https://pypi.org/project/pip/) installed (pip is needed to download dependencies for the role manager Lambda function). ## Instructions diff --git a/docs/infra/set-up-app-monitoring-alerts.md b/docs/infra/set-up-app-monitoring-alerts.md index ee324acb5..3fffaed46 100644 --- a/docs/infra/set-up-app-monitoring-alerts.md +++ b/docs/infra/set-up-app-monitoring-alerts.md @@ -6,10 +6,10 @@ The monitoring module defines metric-based alerting policies that provides aware ## Prerequisites -* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) -* You'll need to have [configured the application](/infra/app/app-config/main.tf) -* You'll need to have [set up the network(s)](./set-up-networks.md) -* If the application needs custom-built container images, you'll need to have [set up the build repository](./set-up-app-build-repository.md) +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You have [configured the application](/infra/app/app-config/main.tf). +* You have [set up the network(s)](./set-up-networks.md). +* If the application needs custom-built container images, you have [set up the build repository](./set-up-app-build-repository.md). ## Instructions diff --git a/docs/infra/set-up-app-service.md b/docs/infra/set-up-app-service.md index e0dca86a4..96a58268f 100644 --- a/docs/infra/set-up-app-service.md +++ b/docs/infra/set-up-app-service.md @@ -12,11 +12,11 @@ The application service setup process will: ## Prerequisites -* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) -* You'll need to have [configured the application](/infra/app/app-config/main.tf) -* You'll need to have [set up the network(s)](./set-up-networks.md) -* Optionally, if you need a container build repository, you'll need to have [set up the build repository](./set-up-app-build-repository.md) -* Optionally, if you need a database for the application, you'll need to have [set up the database](./set-up-app-database.md) +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You have [configured the application](/infra/app/app-config/main.tf). +* You have [set up the network(s)](./set-up-networks.md). +* Optionally, if you need a container build repository, you have [set up the build repository](./set-up-app-build-repository.md). +* Optionally, if you need a database for the application, you have [set up the database](./set-up-app-database.md). ## Instructions diff --git a/docs/infra/set-up-aws-account.md b/docs/infra/set-up-aws-account.md index 53a1a57cb..902af74a6 100644 --- a/docs/infra/set-up-aws-account.md +++ b/docs/infra/set-up-aws-account.md @@ -4,15 +4,15 @@ Follow these instructions for **each AWS account** you want to configure (you ca The AWS account setup process will: -1. Create the [Terraform backend](https://www.terraform.io/language/settings/backends/configuration) resources needed to store Terraform's infrastructure state files. The project uses an [S3 backend](https://www.terraform.io/language/settings/backends/s3). -2. Create the [OpenID connect provider in AWS](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html) to allow GitHub Actions to access AWS account resources. -3. Create the IAM role and policy that GitHub Actions will use to manage infrastructure resources. +* Create the [Terraform backend](https://www.terraform.io/language/settings/backends/configuration) resources needed to store Terraform's infrastructure state files. The project uses an [S3 backend](https://www.terraform.io/language/settings/backends/s3). +* Create the [OpenID connect provider in AWS](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html) to allow GitHub Actions to access AWS account resources. +* Create the IAM role and policy that GitHub Actions will use to manage infrastructure resources. ## Prerequisites -* You'll need to have [set up infrastructure tools](./set-up-infrastructure-tools.md), like Terraform, AWS CLI, and AWS authentication -* You'll also need to make sure the [project is configured](/infra/project-config/main.tf) -* You'll need to have [decided on your environments and AWS account strategy](./set-up-aws-accounts.md) +* You have [set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md). +* You have [configured the project](/infra/project-config/main.tf). +* You have [decided on your environments and AWS account strategy](./set-up-aws-accounts.md). ## Instructions diff --git a/docs/infra/set-up-aws-accounts.md b/docs/infra/set-up-aws-accounts.md index 0d92ba79f..aa0d32459 100644 --- a/docs/infra/set-up-aws-accounts.md +++ b/docs/infra/set-up-aws-accounts.md @@ -2,9 +2,9 @@ The AWS accounts setup process will: -1. Help you decide on your environments -2. Help you decide on your AWS account strategy -3. Configure all of your AWS accounts +* Help you decide on your environments +* Help you decide on your AWS account strategy +* Configure all of your AWS accounts ## Prerequisites diff --git a/docs/infra/set-up-network-custom-domains.md b/docs/infra/set-up-network-custom-domains.md index 203c25131..bf0b1df55 100644 --- a/docs/infra/set-up-network-custom-domains.md +++ b/docs/infra/set-up-network-custom-domains.md @@ -6,16 +6,16 @@ Production systems will want to set up custom domains to route internet traffic The custom domain setup process will: -1. Create an [Amazon Route 53 hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) to manage DNS records for a domain and subdomains -2. Create DNS A (address) records to route traffic from a custom domain to an application's load balancer +* Create an [Amazon Route 53 hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) to manage DNS records for a domain and subdomains +* Create DNS A (address) records to route traffic from a custom domain to an application's load balancer ## Prerequisites -* You'll need to have registered custom domain(s) with a domain registrar (e.g. Namecheap, GoDaddy, Google Domains, etc.) -* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) -* You'll need to have [configured all applications](./set-up-app-config.md) -* You'll need to have [set up the networks](./set-up-network.md) that you want to add the custom domain to -* You'll need to have [set up the application service](./set-up-app-service.md) +* You have registered custom domain(s) with a domain registrar (e.g. Namecheap, GoDaddy, Google Domains, etc.). +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You have [configured all applications](./set-up-app-config.md). +* You have [set up the networks](./set-up-network.md) that you want to add the custom domain to. +* You have [set up the application service](./set-up-app-service.md). ## Instructions diff --git a/docs/infra/set-up-network-https.md b/docs/infra/set-up-network-https.md index adfadbcc5..6372663b6 100644 --- a/docs/infra/set-up-network-https.md +++ b/docs/infra/set-up-network-https.md @@ -4,12 +4,12 @@ Follow these instructions for **each network** (you can have one or more in your Production systems will want to use HTTPS rather than HTTP to prevent man-in-the-middle attacks. This document describes how HTTPS is configured. This process will: -1. Issue an SSL/TLS certificate using Amazon Certificate Manager (ACM) for each domain that we want to support HTTPS -2. Associate the certificate with the application's load balancer so that the load balancer can serve HTTPS requests intended for that domain +* Issue an SSL/TLS certificate using Amazon Certificate Manager (ACM) for each domain that we want to support HTTPS +* Associate the certificate with the application's load balancer so that the load balancer can serve HTTPS requests intended for that domain ## Prerequisites -* You'll need to have [set up custom domains](./set-up-network-custom-domains.md) and met all of those prerequisites +* You have [set up custom domains](./set-up-network-custom-domains.md) and met all of those prerequisites. This is because SSL/TLS certificates must be properly configured for the specific domain to support establishing secure connections. diff --git a/docs/infra/set-up-network.md b/docs/infra/set-up-network.md index f74da8c7d..589fafb57 100644 --- a/docs/infra/set-up-network.md +++ b/docs/infra/set-up-network.md @@ -4,15 +4,15 @@ Follow these instructions for **each network** in your project (you can have one The network setup process will configure and deploy network resources needed by other modules for one network. In particular, it will: -1. Create a nondefault VPC -2. Create public subnets for publicly accessible resources such as the application load balancer, private subnets for the application service, and private subnets for the database. -3. Create VPC endpoints for the AWS services needed by ECS Fargate to fetch the container image and log to AWS CloudWatch. If your application has a database, it will also create VPC endpoints for the AWS services needed by the database layer and a security group to contain those VPC endpoints. +* Create a nondefault VPC +* Create public subnets for publicly accessible resources such as the application load balancer, private subnets for the application service, and private subnets for the database. +* Create VPC endpoints for the AWS services needed by ECS Fargate to fetch the container image and log to AWS CloudWatch. If your application has a database, it will also create VPC endpoints for the AWS services needed by the database layer and a security group to contain those VPC endpoints. ## Prerequisites -* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) -* You'll need to have [configured all applications](./set-up-app-config.md) -* You'll need to have [configured the project's networks](./set-up-networks.md) +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You have [configured all applications](./set-up-app-config.md). +* You have [configured the project's networks](./set-up-networks.md). ## Instructions diff --git a/docs/infra/set-up-networks.md b/docs/infra/set-up-networks.md index 59fa8dc04..92595ae8a 100644 --- a/docs/infra/set-up-networks.md +++ b/docs/infra/set-up-networks.md @@ -2,13 +2,13 @@ The networks set up process will: -1. Configure project networks, including to be compatible with the project's environment mapping -3. Configure all networks +* Configure project networks, including to be compatible with the project's environment mapping +* Configure all networks ## Prerequisites -* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) -* You'll need to have [configured all applications](./set-up-app-config.md) +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You have [configured all applications](./set-up-app-config.md). ## Instructions From 7fe9315973d4303f99bb33be16b8972c54fbba21 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 15:33:14 -0700 Subject: [PATCH 65/89] Improve technical writing --- docs/infra/set-up-aws-account.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/infra/set-up-aws-account.md b/docs/infra/set-up-aws-account.md index 902af74a6..aab6275e0 100644 --- a/docs/infra/set-up-aws-account.md +++ b/docs/infra/set-up-aws-account.md @@ -4,7 +4,7 @@ Follow these instructions for **each AWS account** you want to configure (you ca The AWS account setup process will: -* Create the [Terraform backend](https://www.terraform.io/language/settings/backends/configuration) resources needed to store Terraform's infrastructure state files. The project uses an [S3 backend](https://www.terraform.io/language/settings/backends/s3). +* Create the [Terraform backend](https://www.terraform.io/language/settings/backends/configuration) resources needed to store Terraform's infrastructure state files using an [S3 backend](https://www.terraform.io/language/settings/backends/s3). * Create the [OpenID connect provider in AWS](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html) to allow GitHub Actions to access AWS account resources. * Create the IAM role and policy that GitHub Actions will use to manage infrastructure resources. @@ -12,19 +12,19 @@ The AWS account setup process will: * You have [set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md). * You have [configured the project](/infra/project-config/main.tf). -* You have [decided on your environments and AWS account strategy](./set-up-aws-accounts.md). +* You have [decided on your environment and AWS account strategy](./set-up-aws-accounts.md). ## Instructions ### 1. Make sure you're authenticated into the AWS account you want to configure -The account set up sets up whatever account you're authenticated into. To see which account that is, run +This setup applies to the account you're authenticated into. To see which account that is, run: ```bash aws sts get-caller-identity ``` -To see a more human readable account alias instead of the account, run +To see a more human readable account alias instead of the account, run: ```bash aws iam list-account-aliases @@ -32,7 +32,7 @@ aws iam list-account-aliases ### 2. Create backend resources and tfbackend config file -Run the following command, replacing `` with a human readable name for the AWS account that you're authenticated into. The account name will be used to prefix the created tfbackend file so that it's easier to visually identify as opposed to identifying the file using the account id. For example, you have an account per environment, the account name can be the name of the environment (e.g. "prod" or "staging"). Or if you are setting up an account for all lower environments, account name can be "lowers". If your AWS account has an account alias, you can also use that. +Run the following command, replacing `` with a human readable name for the AWS account that you're authenticated into. The account name will be used to prefix the tfbackend file. For example, if you have an account per environment, the account name can be the name of the environment (e.g. "prod" or "staging"). Or if you are setting up an account for all lower environments, the account name can be "lowers". If your AWS account has an account alias, you can also use that. ```bash make infra-set-up-account ACCOUNT_NAME= @@ -40,9 +40,9 @@ make infra-set-up-account ACCOUNT_NAME= This command will create the S3 tfstate bucket and the GitHub OIDC provider. It will also create a `[account name].[account id].s3.tfbackend` file in the `infra/accounts` directory. -## Making changes to the account +## Making changes to an AWS account -If you make changes to the account Terraform and want to apply those changes, run +If you make changes to an account, apply those changes by running: ```bash make infra-update-current-account From fa18d900236842b428a2d9d5660afeb5b488e4db Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 15:40:39 -0700 Subject: [PATCH 66/89] Improve technical writing --- docs/infra/set-up-app-config.md | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md index a6621a20d..b4dad03e8 100644 --- a/docs/infra/set-up-app-config.md +++ b/docs/infra/set-up-app-config.md @@ -2,12 +2,10 @@ Follow these instructions for **each application** in your project (you can have one or more in your project). -The application config setup process will configure one application. These values will be used in subsequent infra setup steps to determine which resources to deploy and how they will be configured. - ## Prerequisites -* You'll need to have [configured the project](/infra/project-config/main.tf) -* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) +* You have [configured the project](/infra/project-config/main.tf). +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). ## Instructions @@ -15,27 +13,33 @@ The application config setup process will configure one application. These value By default, the application module is named `app` in [`/infra`](/infra/). You may want to rename the application to something project-specific. -### 2. Configure the app-config +### 2. Configure app-config Modify the following values in the application's `app-config/main.tf` (e.g. in `/infra//app-config/main.tf`): * Set the `environments` array to list the names of the environments for this application. By default, this is set to `["dev", "staging", "prod"]`. -* Set `has_database` to `true` or `false` to indicate whether or not the application has a database to integrate with. This setting determines whether or not to create VPC endpoints needed by the database layer and whether. By default, this is set to `false`. +* Set `has_database` to `true` or `false` to indicate whether or not the application relies on a database. This setting determines whether or not to create a database and create VPC endpoints needed by the database layer. By default, this is set to `false`. * Set `has_external_non_aws_service` to `true` or `false` to indicate whether or not your application makes calls to an external non-AWS service. This setting determines whether or not to create NAT gateways, which allows the service in the private subnet to make requests to the internet. By default, this is set to `false`. * Set `has_incident_management_service` to `true` or `false` to indicate whether the application should integrate with an incident management service. By default, this is set to `false`. -* Set the `account_names_by_environment` hash to map environments to AWS accounts. See [set up AWS accounts](./set-up-aws-accounts.md) for more information. +* Set the `account_names_by_environment` hash to map environments to AWS accounts. Use the mapping you decided on in the [set up AWS accounts](./set-up-aws-accounts.md) step. To use [feature flags](/docs/feature-flags.md), modify the values in the application's `app-config/feature-flags.tf` (e.g. in `/infra//app-config/feature-flags.tf`). ### 3. Configure each environment -Within the application's `app-config` directory (e.g. `infra//app-config`), each environment configured in the `environments` array in the previous step should have its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. +Within the application's `app-config` directory (e.g. `infra//app-config`), each environment configured in the `environments` array in the previous step needs its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, there should be corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. In each environment config file, modify the following values: * Set `environment` to the name of the environment. This should match the name of the file. -* Set `network_name`. By default, it should match the name of the environment. This mapping ensures that each network is configured appropriately based on the application(s) in that network (see `local.apps_in_network` in `/infra/networks/main.tf`). Failure to set the network name properly means that the network layer may not receive the correct application configurations for `has_database` and `has_external_non_aws_service`. -* Skip `domain_name` for now -* Skip `enable_https` for now +* Set `network_name`. By default, it should match the name of the environment. This mapping ensures that each network is configured appropriately based on the application(s) in that network (see `local.apps_in_network` in `/infra/networks/main.tf`). Failure to set the network name properly may cause the network layer to use incorrect application configurations for `has_database` and `has_external_non_aws_service`. +* Skip `domain_name` for now. +* Skip `enable_https` for now. + +When configuring the production environment, update these settings based on your project's needs: + +* `service_cpu` +* `service_memory` +* `service_desired_instance_count` -When configuring the production environment, make sure to update the `service_cpu`, `service_memory`, and `service_desired_instance_count` settings based on the project's needs. If your application is sensitive to performance, consider doing a load test. +Consider doing a load test if your application is sensitive to performance. From 4ba3e608736abb35f276989138320c25eb882e93 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 15:42:58 -0700 Subject: [PATCH 67/89] Improve technical writing --- docs/infra/set-up-app-config.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md index b4dad03e8..9668d6184 100644 --- a/docs/infra/set-up-app-config.md +++ b/docs/infra/set-up-app-config.md @@ -15,7 +15,7 @@ By default, the application module is named `app` in [`/infra`](/infra/). You ma ### 2. Configure app-config -Modify the following values in the application's `app-config/main.tf` (e.g. in `/infra//app-config/main.tf`): +Modify the following values in the application's `app-config/main.tf` (e.g. `/infra//app-config/main.tf`): * Set the `environments` array to list the names of the environments for this application. By default, this is set to `["dev", "staging", "prod"]`. * Set `has_database` to `true` or `false` to indicate whether or not the application relies on a database. This setting determines whether or not to create a database and create VPC endpoints needed by the database layer. By default, this is set to `false`. @@ -23,11 +23,11 @@ Modify the following values in the application's `app-config/main.tf` (e.g. in ` * Set `has_incident_management_service` to `true` or `false` to indicate whether the application should integrate with an incident management service. By default, this is set to `false`. * Set the `account_names_by_environment` hash to map environments to AWS accounts. Use the mapping you decided on in the [set up AWS accounts](./set-up-aws-accounts.md) step. -To use [feature flags](/docs/feature-flags.md), modify the values in the application's `app-config/feature-flags.tf` (e.g. in `/infra//app-config/feature-flags.tf`). +To use [feature flags](/docs/feature-flags.md), modify the values in the application's `app-config/feature-flags.tf` (e.g. `/infra//app-config/feature-flags.tf`). ### 3. Configure each environment -Within the application's `app-config` directory (e.g. `infra//app-config`), each environment configured in the `environments` array in the previous step needs its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, there should be corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. +Within the application's `app-config` directory (e.g. `infra//app-config`), each environment configured in the `environments` array in the previous step needs its own config file. For example, if the application has three environments `dev`, `staging`, and `prod`, there must be corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. In each environment config file, modify the following values: From 3341ed4057b63c254175ecb7d3bf2b84e1efb4e0 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 15:48:25 -0700 Subject: [PATCH 68/89] Improve technical writing --- docs/infra/set-up-network.md | 14 +++++++------- docs/infra/set-up-networks.md | 8 ++++---- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/infra/set-up-network.md b/docs/infra/set-up-network.md index 589fafb57..65258d80d 100644 --- a/docs/infra/set-up-network.md +++ b/docs/infra/set-up-network.md @@ -2,10 +2,10 @@ Follow these instructions for **each network** in your project (you can have one or more in your project). -The network setup process will configure and deploy network resources needed by other modules for one network. In particular, it will: +The network set up process will configure and deploy network resources needed for one network. In particular, it will: * Create a nondefault VPC -* Create public subnets for publicly accessible resources such as the application load balancer, private subnets for the application service, and private subnets for the database. +* Create public subnets for publicly accessible resources such as the application load balancer, private subnets for the application service, and private subnets for the database * Create VPC endpoints for the AWS services needed by ECS Fargate to fetch the container image and log to AWS CloudWatch. If your application has a database, it will also create VPC endpoints for the AWS services needed by the database layer and a security group to contain those VPC endpoints. ## Prerequisites @@ -18,13 +18,13 @@ The network setup process will configure and deploy network resources needed by ### 1. Make sure you're authenticated into the AWS account you want to configure -The network is set up for whatever account you're authenticated into. To see which account that is, run +This setup applies to the account you're authenticated into. To see which account that is, run: ```bash aws sts get-caller-identity ``` -To see a more human readable account alias instead of the account, run +To see a more human readable account alias instead of the account, run: ```bash aws iam list-account-aliases @@ -32,7 +32,7 @@ aws iam list-account-aliases ### 2. Configure backend -To create the tfbackend file for the new network, run +To create the `.tfbackend` file for the new network, run: ```bash make infra-configure-network NETWORK_NAME= @@ -40,7 +40,7 @@ make infra-configure-network NETWORK_NAME= ### 3. Create network resources -Now run the following commands to create the resources. Review the Terraform before confirming "yes" to apply the changes. +To create the resources, run the following commend. Review the Terraform output carefully before typing "yes" to apply the changes. ```bash make infra-update-network NETWORK_NAME= @@ -48,4 +48,4 @@ make infra-update-network NETWORK_NAME= ## Updating the network -If you make changes to your application's configuration that impacts the network (such as `has_database` and `has_external_non_aws_service`), make sure to update the network before you update or deploy subsequent infrastructure layers. +If you make changes to your application's configuration that impact the network (such as `has_database` and `has_external_non_aws_service`), update the network before you update subsequent infrastructure layers. diff --git a/docs/infra/set-up-networks.md b/docs/infra/set-up-networks.md index 92595ae8a..d30528280 100644 --- a/docs/infra/set-up-networks.md +++ b/docs/infra/set-up-networks.md @@ -2,25 +2,25 @@ The networks set up process will: -* Configure project networks, including to be compatible with the project's environment mapping +* Configure project networks to be compatible with your project's environment mapping * Configure all networks ## Prerequisites * You have [set up the AWS account(s)](./set-up-aws-accounts.md). -* You have [configured all applications](./set-up-app-config.md). +* You have [configured all application(s)](./set-up-app-config.md). ## Instructions ### 1. Configure the project's networks -Modify [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) to ensure the environments listed match what you decided in the [set up AWS accounts](./set-up-aws-accounts.md) step. +Modify [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) so that the environments listed match what you decided on in the [set up AWS accounts](./set-up-aws-accounts.md) step. By default there are three networks defined, one for each application environment. You can add additional additional networks as desired. If you have multiple applications and want your applications in separate networks within the same AWS account, you may want to give the networks differentiating names (e.g. "foo-dev", "foo-prod", "bar-dev", "bar-prod", instead of just "dev", "prod"). -Skip the `domain_config` config for now. These settings are optionally configured later when [setting up custom domains](./set-up-network-custom-domains.md) and when [setting up HTTPS](./set-up-network-https.md). +Skip the `domain_config` config for now. These settings are configured later when [setting up custom domains](./set-up-network-custom-domains.md) and when [setting up HTTPS](./set-up-network-https.md). ### 2. Set up each network From 7bbe2d131f75b0e53eaff8ebb07cfdd5a829f9ea Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 15:51:01 -0700 Subject: [PATCH 69/89] Improve technical writing --- docs/infra/set-up-app-build-repository.md | 20 ++++++++++---------- docs/infra/set-up-network.md | 2 +- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/infra/set-up-app-build-repository.md b/docs/infra/set-up-app-build-repository.md index a36172376..f044fb28c 100644 --- a/docs/infra/set-up-app-build-repository.md +++ b/docs/infra/set-up-app-build-repository.md @@ -2,25 +2,25 @@ Follow these instructions for **each application** in your project (you can have one or more in your project). If the application does not need a build repository, skip to the bottom of this document. -The application build repository setup process will create infrastructure resources needed to store built release candidate artifacts used to deploy the application to an environment. +The application build repository set up process will create the infrastructure resources needed to store built release candidate artifacts used to deploy the application to an environment. ## Prerequisites -* You'll need to have [set up the AWS account(s)](./set-up-aws-accounts.md) -* You'll need to have [configured the application](/infra/app/app-config/main.tf) -* You'll need to have [set up the network(s)](./set-up-networks.md) +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You have [configured the application](/infra/app/app-config/main.tf). +* You have [set up the network(s)](./set-up-networks.md). ## Instructions ### 1. Make sure you're authenticated into the AWS account where you want to deploy resources shared across environments -This set up takes effect in whatever account you're authenticated into. To see which account that is, run +This setup applies to the account you're authenticated into. To see which account that is, run: ```bash aws sts get-caller-identity ``` -To see a more human readable account alias instead of the account, run +To see a more human readable account alias instead of the account, run: ```bash aws iam list-account-aliases @@ -28,17 +28,17 @@ aws iam list-account-aliases ### 2. Configure backend -To create the tfbackend file for the build repository using the backend configuration values from your current AWS account, run +To create the `.tfbackend` file for the build repository, run: ```bash make infra-configure-app-build-repository APP_NAME= ``` -`APP_NAME` needs to be the name of the application folder within the `infra` folder. +`APP_NAME` should be the name of the application folder within the `infra` folder. ### 3. Create build repository resources -Now run the following commands to create the resources, making sure to verify the plan before confirming the apply. +To create the resources, run the following command. Review the Terraform output carefully before typing "yes" to apply the changes. ```bash make infra-update-app-build-repository APP_NAME= @@ -46,4 +46,4 @@ make infra-update-app-build-repository APP_NAME= ## If the application does not need a build repository -If the application does not need a build repository (such as if the project uses pre-built images hosted in an external container repository), delete the application's build repository module (e.g. `/infra//build-repository`). \ No newline at end of file +If the application does not need a build repository, such as if the project uses pre-built images hosted in an external container repository, delete the application's build repository module (e.g. `/infra//build-repository`). \ No newline at end of file diff --git a/docs/infra/set-up-network.md b/docs/infra/set-up-network.md index 65258d80d..5019128ff 100644 --- a/docs/infra/set-up-network.md +++ b/docs/infra/set-up-network.md @@ -40,7 +40,7 @@ make infra-configure-network NETWORK_NAME= ### 3. Create network resources -To create the resources, run the following commend. Review the Terraform output carefully before typing "yes" to apply the changes. +To create the resources, run the following command. Review the Terraform output carefully before typing "yes" to apply the changes. ```bash make infra-update-network NETWORK_NAME= From 460aac56c1d00fb1e402ec4abaa89b72c4a484d9 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 15:57:03 -0700 Subject: [PATCH 70/89] Improve technical writing --- README.md | 2 ++ docs/infra/set-up-app-database.md | 26 ++++++++++++++------------ 2 files changed, 16 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index fa8ead7f8..82626757c 100644 --- a/README.md +++ b/README.md @@ -64,7 +64,9 @@ By default, the update script will apply changes from the `main` branch of this ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- ``` + `` should be the version of the template to install. This can be a branch, commit hash, or tag. + `` should be the type of `` provided. Defaults to `branch`. This can be: `branch`, `commit`, or `tag`. Examples: diff --git a/docs/infra/set-up-app-database.md b/docs/infra/set-up-app-database.md index 434d3cc39..fb0ada886 100644 --- a/docs/infra/set-up-app-database.md +++ b/docs/infra/set-up-app-database.md @@ -2,13 +2,13 @@ Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. If the application does not need a database, skip to the bottom of this document. -The database setup process will: +The database set up process will: * Configure and deploy an application database cluster using [Amazon Aurora Serverless V2](https://aws.amazon.com/rds/aurora/serverless/) -* Create a [PostgreSQL schema](https://www.postgresql.org/docs/current/ddl-schemas.html) `app` to contain tables used by the application +* Create a [PostgreSQL schema](https://www.postgresql.org/docs/current/ddl-schemas.html) called `app` to contain tables used by the application * Create an IAM policy that allows IAM roles with that policy attached to [connect to the database using IAM authentication](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.html) -* Create an [AWS Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html), the "role manager", for provisioning the [PostgreSQL database users](https://www.postgresql.org/docs/8.0/user-manag.html) that will be used by the application service and by the migrations task -* Invoke the role manager function to create the `app` and `migrator` Postgres users +* Create an [AWS Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) called "role manager" to provision the [PostgreSQL database users](https://www.postgresql.org/docs/8.0/user-manag.html) used by the application service and the migrations task +* Invoke the role manager function to create Postgres users called `app` and `migrator` ## Prerequisites @@ -21,13 +21,13 @@ The database setup process will: ### 1. Make sure you're authenticated into the AWS account where you want to deploy this environment -This set up takes effect in whatever account you're authenticated into. To see which account that is, run +This setup applies to the account you're authenticated into. To see which account that is, run: ```bash aws sts get-caller-identity ``` -To see a more human readable account alias instead of the account, run +To see a more human readable account alias instead of the account, run: ```bash aws iam list-account-aliases @@ -35,19 +35,19 @@ aws iam list-account-aliases ### 2. Configure backend -To create the tfbackend file for the new application environment, run +To create the `.tfbackend` file for the new application environment, run: ```bash make infra-configure-app-database APP_NAME= ENVIRONMENT= ``` -`APP_NAME` needs to be the name of the application folder within the `infra` folder. +`APP_NAME` should be the name of the application folder within the `infra` folder. -`ENVIRONMENT` needs to be the name of the environment to update. This will create a file called `.s3.tfbackend` in the `infra//database` module directory. +`ENVIRONMENT` should be the name of the environment to update. This will create a file called `.s3.tfbackend` in `infra//database`. ### 3. Create database resources -Now run the following commands to create the resources. Review the Terraform before confirming "yes" to apply the changes. This can take over 5 minutes. +To create the resources, run the following command. Review the Terraform output carefully before typing "yes" to apply the changes. This can take over 5 minutes. ```bash make infra-update-app-database APP_NAME= ENVIRONMENT= @@ -55,7 +55,7 @@ make infra-update-app-database APP_NAME= ENVIRONMENT= ### 4. Create Postgres users -Trigger the role manager Lambda function that was created in the previous step in order to create the application and migrator Postgres users. +To trigger the role manager Lambda function that was created in the previous step to create the application and migrator Postgres users, run: ```bash make infra-update-app-database-roles APP_NAME= ENVIRONMENT= @@ -98,10 +98,12 @@ Why is this needed? The reason is because the `migrator` role will be used by th ### 5. Check that database roles have been configured properly +Verify the that the database roles have been configured properly by running: + ```bash make infra-check-app-database-roles APP_NAME= ENVIRONMENT= ``` ## If the application does not need a database -If the application does not need a database (such as if the project uses an alternative for data persistence), delete the application's database module (e.g. `/infra//database`) and ensure that the application's `app-config` sets `has_database` to `false` (see [set up app config](./set-up-app-config.md)). \ No newline at end of file +If the application does not need a database, such as if the project uses an alternative for data persistence, delete the application's database module (e.g. `/infra//database`) and set the application's `app-config` setting `has_database` to `false` (see [set up app config](./set-up-app-config.md)). \ No newline at end of file From bf5466e8b8ffd1fa592c54e3ac1b9ff28cb08c14 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 16:00:33 -0700 Subject: [PATCH 71/89] Improve technical writing --- docs/infra/set-up-app-service.md | 37 +++++++++++++++----------------- 1 file changed, 17 insertions(+), 20 deletions(-) diff --git a/docs/infra/set-up-app-service.md b/docs/infra/set-up-app-service.md index 96a58268f..1502a79f0 100644 --- a/docs/infra/set-up-app-service.md +++ b/docs/infra/set-up-app-service.md @@ -2,13 +2,13 @@ Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. -The application service setup process will: +The application service set up process will: * Configure an ECS Fargate Service and Task to host the application -* Creates a load balancer for the application +* Create a load balancer for the application * Create an S3 bucket for general object storage * Set up CloudWatch for logging, monitoring, and alerts -* Configure CloudWatch Evidently to support [feature flags](/docs/feature-flags.md) +* Configure CloudWatch Evidently to support feature flags ## Prerequisites @@ -22,13 +22,13 @@ The application service setup process will: ### 1. Make sure you're authenticated into the AWS account where you want to deploy this environment -This set up takes effect in whatever account you're authenticated into. To see which account that is, run +This setup applies to the account you're authenticated into. To see which account that is, run: ```bash aws sts get-caller-identity ``` -To see a more human readable account alias instead of the account, run +To see a more human readable account alias instead of the account, run: ```bash aws iam list-account-aliases @@ -36,35 +36,32 @@ aws iam list-account-aliases ### 2. Configure backend -To create the tfbackend and tfvars files for the new application service, run +To create the `.tfbackend` and `.tfvars` files for the new application service, run: ```bash make infra-configure-app-service APP_NAME= ENVIRONMENT= ``` -`APP_NAME` needs to be the name of the application folder within the `infra` folder. +`APP_NAME` must be the name of the application folder within the `infra` folder. -`ENVIRONMENT` needs to be the name of the environment to update. This will create a file called `.s3.tfbackend` in the `infra//service` module directory. +`ENVIRONMENT` must be the name of the environment to update. This will create a file called `.s3.tfbackend` in `infra//service`. -### 3. Build and publish the application to the application build repository +### 3. Build and publish the application to the build repository -Before creating the application resources, you'll need to first build and publish at least one image to the application build repository. This step does not need to be run per-environment. +Before creating the application resources, you need to build and publish at least one image to the build repository. This step does not need to be run per environment. -There are two ways to do this: +Run the following commands from the project's root directory: -1. Trigger the "Build and Publish" workflow from your repo's GitHub Actions tab. This option requires that the `role-to-assume` GitHub workflow variable has already been setup as part of the overall infra account setup process. -1. Alternatively, run the following from the root directory. This option can take much longer than the GitHub workflow, depending on your machine's architecture. - - ```bash - make release-build APP_NAME= - make release-publish APP_NAME= - ``` +```bash +make release-build APP_NAME= +make release-publish APP_NAME= +``` -Copy the image tag name that was published. You'll need this in the next step. +Copy the image tag name that is output. You'll need this in the next step. ### 4. Create application resources with the image tag that was published -Now run the following commands to create the resources, using the image tag that was published from the previous step. Review the Terraform before confirming "yes" to apply the changes. +To create the resources, run the following command using the image tag from the previous step. Review the Terraform output carefully before typing "yes" to apply the changes. This can take over 5 minutes. ```bash TF_CLI_ARGS_apply="-var=image_tag=" make infra-update-app-service APP_NAME= ENVIRONMENT= From 5c3f37dd2f5a3451519a85b2068af0642df293c6 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 16:02:52 -0700 Subject: [PATCH 72/89] Improve technical writing --- docs/infra/set-up-app-monitoring-alerts.md | 18 ++++++++---------- docs/infra/set-up-app-service.md | 4 ++-- 2 files changed, 10 insertions(+), 12 deletions(-) diff --git a/docs/infra/set-up-app-monitoring-alerts.md b/docs/infra/set-up-app-monitoring-alerts.md index 3fffaed46..9a2232c2d 100644 --- a/docs/infra/set-up-app-monitoring-alerts.md +++ b/docs/infra/set-up-app-monitoring-alerts.md @@ -9,15 +9,13 @@ The monitoring module defines metric-based alerting policies that provides aware * You have [set up the AWS account(s)](./set-up-aws-accounts.md). * You have [configured the application](/infra/app/app-config/main.tf). * You have [set up the network(s)](./set-up-networks.md). -* If the application needs custom-built container images, you have [set up the build repository](./set-up-app-build-repository.md). +* If you need a container build repository, you have [set up the build repository](./set-up-app-build-repository.md). -## Instructions - -### To set up email alerts +## Instructions to set up email alerts When any of the alerts described by the module are triggered, a notification will be send to all emails specified in the `email_alerts_subscription_list`. -#### 1. Update the application's service layer +### 1. Update the application's service layer In the application's service module (e.g. `/infra//service/main.tf`), uncomment the `email_alerts_subscription_list` key and add the emails that should be notified. @@ -31,7 +29,7 @@ module "monitoring" { } ``` -#### 2. Update the application service +### 2. Update the application service To apply the changes, run the following command for each environment: @@ -43,13 +41,13 @@ make infra-update-app-service APP_NAME= ENVIRONMENT= `ENVIRONMENT` needs to be the name of the environment to update. -### To set up external incident management service integration +## Instructions to set up external incident management service integration -#### 1. Update the application config +### 1. Update the application config Modify the `has_incident_management_service` in the application's `app-config/main.tf` (e.g. in `/infra//app-config/main.tf`) and set it to `true`. -#### 2. Add the external url as a secret +### 2. Add the external url as a secret Get the integration URL for the incident management service and store it in AWS SSM Parameter Store by running the following command @@ -57,7 +55,7 @@ Get the integration URL for the incident management service and store it in AWS make infra-configure-monitoring-secrets APP_NAME= ENVIRONMENT= URL= ``` -#### 3. Update the application service +### 3. Update the application service To apply the changes, run the following command diff --git a/docs/infra/set-up-app-service.md b/docs/infra/set-up-app-service.md index 1502a79f0..290ff5496 100644 --- a/docs/infra/set-up-app-service.md +++ b/docs/infra/set-up-app-service.md @@ -15,8 +15,8 @@ The application service set up process will: * You have [set up the AWS account(s)](./set-up-aws-accounts.md). * You have [configured the application](/infra/app/app-config/main.tf). * You have [set up the network(s)](./set-up-networks.md). -* Optionally, if you need a container build repository, you have [set up the build repository](./set-up-app-build-repository.md). -* Optionally, if you need a database for the application, you have [set up the database](./set-up-app-database.md). +* If you need a container build repository, you have [set up the build repository](./set-up-app-build-repository.md). +* If you need a database for the application, you have [set up the database](./set-up-app-database.md). ## Instructions From 156ff858fe4a9fe6ae0337352502c06a6658e067 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 16:06:02 -0700 Subject: [PATCH 73/89] Improve technical writing --- docs/infra/set-up-app-monitoring-alerts.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/infra/set-up-app-monitoring-alerts.md b/docs/infra/set-up-app-monitoring-alerts.md index 9a2232c2d..5b6abc613 100644 --- a/docs/infra/set-up-app-monitoring-alerts.md +++ b/docs/infra/set-up-app-monitoring-alerts.md @@ -13,7 +13,7 @@ The monitoring module defines metric-based alerting policies that provides aware ## Instructions to set up email alerts -When any of the alerts described by the module are triggered, a notification will be send to all emails specified in the `email_alerts_subscription_list`. +When any of the alerts described by the module are triggered, a notification will be sent to all emails specified in `email_alerts_subscription_list`. ### 1. Update the application's service layer @@ -31,25 +31,25 @@ module "monitoring" { ### 2. Update the application service -To apply the changes, run the following command for each environment: +To apply the changes, run the following command. Review the Terraform output carefully before typing "yes" to apply the changes. ```bash make infra-update-app-service APP_NAME= ENVIRONMENT= ``` -`APP_NAME` needs to be the name of the application folder within the `infra` folder. +`APP_NAME` must be the name of the application folder within the `infra` folder. -`ENVIRONMENT` needs to be the name of the environment to update. +`ENVIRONMENT` must be the name of the environment to update. ## Instructions to set up external incident management service integration ### 1. Update the application config -Modify the `has_incident_management_service` in the application's `app-config/main.tf` (e.g. in `/infra//app-config/main.tf`) and set it to `true`. +In the application's `app-config/main.tf` (e.g. `/infra//app-config/main.tf`), set `has_incident_management_service` to `true`. ### 2. Add the external url as a secret -Get the integration URL for the incident management service and store it in AWS SSM Parameter Store by running the following command +Get the integration URL for the incident management service and run the following command to store it in AWS SSM Parameter Store: ```bash make infra-configure-monitoring-secrets APP_NAME= ENVIRONMENT= URL= @@ -57,7 +57,7 @@ make infra-configure-monitoring-secrets APP_NAME= ENVIRONMENT= ENVIRONMENT= @@ -65,7 +65,7 @@ make infra-update-app-service APP_NAME= ENVIRONMENT= ## If the application does not need monitoring notifications -If the application does not need monitoring notifications, ensure that: +If the application does not need monitoring notifications, complete the following steps: -* the application's service module (e.g. `/infra//service/main.tf`) has the `email_alerts_subscription_list` commented out -* the application's `app-config` (e.g. in `/infra//app-config/main.tf`) has `has_incident_management_service` set to `false` \ No newline at end of file +* In the application's service module (e.g. `/infra//service/main.tf`), comment out `email_alerts_subscription_list`. +* In the application's `app-config` (e.g. `/infra//app-config/main.tf`), set `has_incident_management_service` to `false`. \ No newline at end of file From ea81834edf95ef5693c51f2ea048a1ec2850b726 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 16:06:32 -0700 Subject: [PATCH 74/89] Improve technical writing --- docs/infra/set-up-app-monitoring-alerts.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/infra/set-up-app-monitoring-alerts.md b/docs/infra/set-up-app-monitoring-alerts.md index 5b6abc613..5c419bf32 100644 --- a/docs/infra/set-up-app-monitoring-alerts.md +++ b/docs/infra/set-up-app-monitoring-alerts.md @@ -67,5 +67,5 @@ make infra-update-app-service APP_NAME= ENVIRONMENT= If the application does not need monitoring notifications, complete the following steps: -* In the application's service module (e.g. `/infra//service/main.tf`), comment out `email_alerts_subscription_list`. -* In the application's `app-config` (e.g. `/infra//app-config/main.tf`), set `has_incident_management_service` to `false`. \ No newline at end of file +1. In the application's service module (e.g. `/infra//service/main.tf`), comment out `email_alerts_subscription_list`. +2. In the application's `app-config` (e.g. `/infra//app-config/main.tf`), set `has_incident_management_service` to `false`. \ No newline at end of file From 44486a122302b9e54ba7527db7b60817dc35e44a Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 16:14:40 -0700 Subject: [PATCH 75/89] Improve technical writing --- docs/infra/set-up-network-custom-domains.md | 34 ++++++++++----------- docs/infra/set-up-network.md | 2 +- 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/docs/infra/set-up-network-custom-domains.md b/docs/infra/set-up-network-custom-domains.md index bf0b1df55..0fbc071a5 100644 --- a/docs/infra/set-up-network-custom-domains.md +++ b/docs/infra/set-up-network-custom-domains.md @@ -2,9 +2,9 @@ Follow these instructions for **each network** (you can have one or more in your project) in your project. If the network or an application does not need custom domains, skip to the bottom of this document. -Production systems will want to set up custom domains to route internet traffic to their application services rather than using AWS-generated hostnames for the load balancers or the CDN. This document describes how to configure custom domains. +Production systems typically use custom domains to route internet traffic to their application services instead of AWS-generated hostnames for the load balancers or the CDN. -The custom domain setup process will: +The custom domain set up process will: * Create an [Amazon Route 53 hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) to manage DNS records for a domain and subdomains * Create DNS A (address) records to route traffic from a custom domain to an application's load balancer @@ -13,7 +13,7 @@ The custom domain setup process will: * You have registered custom domain(s) with a domain registrar (e.g. Namecheap, GoDaddy, Google Domains, etc.). * You have [set up the AWS account(s)](./set-up-aws-accounts.md). -* You have [configured all applications](./set-up-app-config.md). +* You have [configured all application(s)](./set-up-app-config.md). * You have [set up the networks](./set-up-network.md) that you want to add the custom domain to. * You have [set up the application service](./set-up-app-service.md). @@ -21,13 +21,13 @@ The custom domain setup process will: ### 1. Make sure you're authenticated into the AWS account you want to configure -This set up takes effect in whatever account you're authenticated into. To see which account that is, run +This setup applies to the account you're authenticated into. To see which account that is, run: ```bash aws sts get-caller-identity ``` -To see a more human readable account alias instead of the account, run +To see a more human readable account alias instead of the account, run: ```bash aws iam list-account-aliases @@ -35,14 +35,12 @@ aws iam list-account-aliases ### 2. Set hosted zone in domain configuration -The custom domain configuration is defined as a `domain_config` object in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf). A [hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) represents a domain and all of its subdomains. - -For example, a hosted zone of `platform-test.navateam.com` includes `platform-test.navateam.com`, `cdn.platform-test.navateam.com`, `notifications.platform-test.navateam.com`, `foo.bar.platform-test.navateam.com`, etc. +The custom domain configuration is defined as a `domain_config` object in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf). A [hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) represents a domain and all of its subdomains. For example, a hosted zone of `platform-test.navateam.com` includes `platform-test.navateam.com`, `cdn.platform-test.navateam.com`, `notifications.platform-test.navateam.com`, `foo.bar.platform-test.navateam.com`, etc. **For each network** you want to use a custom domain, in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf): -* set the `hosted_zone` to match the custom domain (or a subdomain of the custom domain) that you registered -* set `manage_dns` to `true` +1. Set the `hosted_zone` to match the custom domain (or a subdomain of the custom domain) that you registered. +2. Set `manage_dns` to `true`. ### 3. Update the network layer to create the hosted zones @@ -54,7 +52,9 @@ make infra-update-network NETWORK_NAME= ### 4. Delegate DNS requests to the newly created hosted zone -You most likely registered your domain outside of this project. Using whichever service you used to register the domain name (e.g. Namecheap, GoDaddy, Google Domains, etc.), add a DNS NS (nameserver) record. Set the "name" equal to the `hosted_zone` and set the value equal to the list of hosted zone name servers that was created in the previous step. You can see the list of servers by running +You most likely registered your domain outside of this project. Using whichever service you used to register the domain name (e.g. Namecheap, GoDaddy, Google Domains, etc.), add a DNS NS (nameserver) record. Set the "name" equal to the `hosted_zone` and set the value equal to the list of hosted zone name servers that was created in the previous step. + +Output a list of servers by running: ```bash terraform -chdir=infra/networks output -json hosted_zone_name_servers @@ -77,7 +77,7 @@ ns-687.awsdns-21.net. ns-80.awsdns-10.com. ``` -Run the following command to verify that DNS requests are being served by the hosted zone nameservers using `nslookup`. +Verify that DNS requests are being served by the hosted zone nameservers by running the following command: ```bash nslookup -type=NS @@ -95,19 +95,19 @@ The `domain_name` must be either the same as the `hosted_zone` or a subdomain of ### 6. Update the application service -**For each application and each environment** in the network that should use the custom domain, apply the changes with the following command +**For each application and each environment** in the network that should use the custom domain, apply the changes by running the following command. Review the Terraform output carefully before typing "yes" to apply the changes. ```bash make infra-update-app-service APP_NAME= ENVIRONMENT= ``` -`APP_NAME` needs to be the name of the application folder within the `infra` folder. +`APP_NAME` must be the name of the application folder within the `infra` folder. -`ENVIRONMENT` needs to be the name of the environment to update. +`ENVIRONMENT` must be the name of the environment to update. ## If a network does not need custom domains -For each network that does not need custom domains, ensure the network's `domain_config` setting in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) looks like this: +For each network that does not need custom domains, set the network's `domain_config` object in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) to the following: ```hcl domain_config = { @@ -119,4 +119,4 @@ domain_config = { ## If an application does not need custom domains -For each application that does not need custom domains, ensure that the application's `app-config/.tf` file (e.g. in `/infra//app-config/.tf`) has `domain_name` set to `""` (empty string). +For each application that does not need custom domains, in the application's `app-config/.tf` file (e.g. `/infra//app-config/.tf`), set `domain_name` to `""` (empty string). diff --git a/docs/infra/set-up-network.md b/docs/infra/set-up-network.md index 5019128ff..f935c3e63 100644 --- a/docs/infra/set-up-network.md +++ b/docs/infra/set-up-network.md @@ -11,7 +11,7 @@ The network set up process will configure and deploy network resources needed fo ## Prerequisites * You have [set up the AWS account(s)](./set-up-aws-accounts.md). -* You have [configured all applications](./set-up-app-config.md). +* You have [configured all application(s)](./set-up-app-config.md). * You have [configured the project's networks](./set-up-networks.md). ## Instructions From a820bf56f57b9eec29f60074374eb0028218a52d Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 16:20:03 -0700 Subject: [PATCH 76/89] Improve technical writing --- docs/infra/set-up-network-https.md | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/docs/infra/set-up-network-https.md b/docs/infra/set-up-network-https.md index 6372663b6..95dfefff8 100644 --- a/docs/infra/set-up-network-https.md +++ b/docs/infra/set-up-network-https.md @@ -2,10 +2,12 @@ Follow these instructions for **each network** (you can have one or more in your project) in your project. If the network or an application does not need HTTPS, skip to the bottom of this document. -Production systems will want to use HTTPS rather than HTTP to prevent man-in-the-middle attacks. This document describes how HTTPS is configured. This process will: +To prevent man-in-the-middle attacks, production systems should use HTTPS rather than HTTP. -* Issue an SSL/TLS certificate using Amazon Certificate Manager (ACM) for each domain that we want to support HTTPS -* Associate the certificate with the application's load balancer so that the load balancer can serve HTTPS requests intended for that domain +The HTTPS set up process will: + +* Issue an SSL/TLS certificate using Amazon Certificate Manager (ACM) for each domain that should support HTTPS +* Associate the certificate with the application's load balancer, so that the load balancer can serve HTTPS requests ## Prerequisites @@ -17,13 +19,13 @@ This is because SSL/TLS certificates must be properly configured for the specifi ### 1. Make sure you're authenticated into the AWS account you want to configure -This set up takes effect in whatever account you're authenticated into. To see which account that is, run +This setup applies to the account you're authenticated into. To see which account that is, run: ```bash aws sts get-caller-identity ``` -To see a more human readable account alias instead of the account, run +To see a more human readable account alias instead of the account, run: ```bash aws iam list-account-aliases @@ -37,13 +39,13 @@ Set the `source` of the domain or subdomain to `issued`. ### 3. Update the network layer to issue the certificates -**For each network** you configured in the previous step, run the following command to issue SSL/TLS certificates +**For each network** you configured in the previous step, apply the changes by running the following command. Review the Terraform output carefully before typing "yes" to apply the changes. This will issue SSL/TLS certificates. ```bash make infra-update-network NETWORK_NAME= ``` -Run the following command to check the status of a certificate (replace `` using the output from the previous command): +Run the following command to check the status of a certificate (replace `` with the output from the previous command): ```bash aws acm describe-certificate --certificate-arn --query Certificate.Status @@ -61,24 +63,24 @@ You should have already set `domain_name` as part of [setting up custom domain n ### 5. Attach certificate to load balancer -**For each application and environment** that should use HTTPS, apply the changes in the previous step by running +**For each application and environment** that should use HTTPS, apply the changes from the previous step by running the following command. Review the Terraform output carefully before typing "yes" to apply the changes. ```bash make infra-update-app-service APP_NAME= ENVIRONMENT= ``` -`APP_NAME` needs to be the name of the application folder within the `infra` folder. +`APP_NAME` must be the name of the application folder within the `infra` folder. -`ENVIRONMENT` needs to be the name of the environment to update. +`ENVIRONMENT` must be the name of the environment to update. ## If a network does not need HTTPS **⚠️ This is not advised** for any network containing a production environment. -For each network that does not need custom domains, ensure the network's `certificate_configs` setting in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) is `{}` (empty hash). +For each network that does not need custom domains, in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf), set the network's `certificate_configs` to `{}` (empty hash). ## If an application does not need HTTPS **⚠️ This is not advised** for an application deployed to a production environment. -For each application that does not need HTTPS, ensure that the application's `app-config/.tf` file (e.g. in `/infra//app-config/.tf`) has `enable_https` set to `false`. \ No newline at end of file +For each application that does not need HTTPS, in the application's `app-config/.tf` file (e.g. in `/infra//app-config/.tf`), set `enable_https` to `false`. \ No newline at end of file From 98df1cb2e40ce0c0e0924327c21f2d8379c35024 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 16:27:27 -0700 Subject: [PATCH 77/89] Improve technical writing --- README.md | 38 +---------------- docs/infra/set-up-infrastructure-tools.md | 4 ++ template-only-docs/multiple-applications.md | 16 +++---- template-only-docs/update-template.md | 46 +++++++++++++++++++++ 4 files changed, 59 insertions(+), 45 deletions(-) create mode 100644 template-only-docs/update-template.md diff --git a/README.md b/README.md index 82626757c..fd904ce32 100644 --- a/README.md +++ b/README.md @@ -49,40 +49,4 @@ After downloading and installing this template into your project, take the follo ## Updates -This template includes a script to update your project to a newer version of the template. The [update script](/template-only-bin/update-template.sh) assumes that your project is version-controlled using `git`. The script will edit your project files, but it will not run `git commit`. After running the script, use `git diff` to review all changes carefully. - -To update your project to a newer version of this template, run the following command in your project's root directory: - -```bash -curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- -``` - -`` is a required argument. It must be a comma-separated list (no spaces) of the apps in `/infra`. App names must be hyphen-separated (i.e. kebab-case). Examples: `app`, `app,app2`, `my-app,your-app`. - -By default, the update script will apply changes from the `main` branch of this template repo. If you want to update to a different branch, a specific commit, or a specific tag (e.g. a release tag), run this instead: - -```bash -curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- -``` - -`` should be the version of the template to install. This can be a branch, commit hash, or tag. - -`` should be the type of `` provided. Defaults to `branch`. This can be: `branch`, `commit`, or `tag`. - -Examples: - -* If your project has one application named `app` and you want to update it to the `main` branch of this template repo, run: - ```bash - curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- app - ``` -* If your project has two applications named `app, app2` and you want to update to the commit `d42963d007e55cc37ef666019428b1d06a25cf71`, run: - ```bash - curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- app,app2 d42963d007e55cc37ef666019428b1d06a25cf71 commit - ``` - -* If your project has three applications named `foo,bar,baz` and you want to update to the `v.0.8.0` release tag, run: - ```bash - curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- foo,bar,baz v0.8.0 tag - ``` - -**Remember:** Read the release notes in case there are breaking changes you need to address. +This template includes a script to update your project to a newer version of the template. To update your project to a newer version of this template, follow the [update template instructions](/template-only-docs/update-template.md). \ No newline at end of file diff --git a/docs/infra/set-up-infrastructure-tools.md b/docs/infra/set-up-infrastructure-tools.md index b939b4a39..a16c8745b 100644 --- a/docs/infra/set-up-infrastructure-tools.md +++ b/docs/infra/set-up-infrastructure-tools.md @@ -2,6 +2,10 @@ Complete these steps to work on the infrastructure. +## Prerequisites + +* None + ## Instructions ### Install Terraform diff --git a/template-only-docs/multiple-applications.md b/template-only-docs/multiple-applications.md index a434efe2e..e6ff6948d 100644 --- a/template-only-docs/multiple-applications.md +++ b/template-only-docs/multiple-applications.md @@ -1,32 +1,32 @@ # Multiple Applications -This infrastructure supports deployment and CI/CD for projects with multiple applications. By default, the infrastructure assumes the project only has one application, named `app`. However, it's straightforward to include additional applications. +You can use this template with multiple applications. By default, this template assumes your project has one application named `app`. However, it's straightforward to add additional applications. ## Prerequisites * None -## Instructions +## Instructions to add an additional application ### 1. Ensure the application meets the Application Requirements -In order to use this infrastructure, the application must meets the [application requirements](/template-only-docs/application-requirements.md). +Applications must meet [these requirements](/template-only-docs/application-requirements.md) to be used with this template. If you're using a [Platform application template](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), these requirements are already met. ### 2. Add the application to the root directory -Add the application's source code to a folder that lives in the project root folder, such as `/second-app`. +Add the application's source code to a folder in the project's root folder (e.g. `/second-app`). -⚠️ Warning: In general, it's best to use a short, descriptive one-word name because some AWS resources have character limits. If you must use multiple words, use hyphens (not underscores) to separate each word. +⚠️ Warning: In general, it's best to use short, descriptive one-word names because some AWS resources have character limits. If you must use multiple words, use hyphens (not underscores) to separate each word. ### 3. Add infrastructure support for the application -Run the following to install the application +To add the Terraform modules for the application, run: ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/download-and-install-app.sh | bash -s -- ``` -`` needs to be the name of the application you chose in the previous step. +`` must be the name of the application you chose in the previous step. This will add a new terraform module at `/infra/` and create the following CI/CD workflows: @@ -35,4 +35,4 @@ This will add a new terraform module at `/infra/` and create the follo ### 4. Configure the application as usual -Follow the per-application steps in [`/infra/README.md`](/infra/README.md) to configure the application. \ No newline at end of file +Follow the per-application steps in [`/infra/README.md`](/infra/README.md) to complete setup. \ No newline at end of file diff --git a/template-only-docs/update-template.md b/template-only-docs/update-template.md new file mode 100644 index 000000000..f0593baf4 --- /dev/null +++ b/template-only-docs/update-template.md @@ -0,0 +1,46 @@ +# Updates + +## Prerequisites + +* The [update script](/template-only-bin/update-template.sh) assumes that your project is version-controlled using `git`. The script will edit your project files, but it will not run `git commit`. After running the script, use `git diff` to review all changes carefully. + +## Instructions + +To update your project to a newer version of this template, run the following command in your project's root directory: + +```bash +curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- +``` + +`` is a required argument. It must be a comma-separated list (no spaces) of the apps in `/infra`. App names must be hyphen-separated (i.e. kebab-case). Examples: `app`, `app,app2`, `my-app,your-app`. + +**Remember:** Read the release notes in case there are breaking changes you need to address. + +### Specifying a branch, a commit, or a tag + +By default, the update script will apply changes from the `main` branch of this template repo. If you want to update to a different branch, a specific commit, or a specific tag (e.g. a release tag), run this instead: + +```bash +curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- +``` + +`` should be the version of the template to install. This can be a branch, commit hash, or tag. + +`` should be the type of `` provided. Defaults to `branch`. This can be: `branch`, `commit`, or `tag`. + +### Examples + +If your project has one application named `app` and you want to update it to the `main` branch of this template repo, run: +```bash +curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- app +``` + +If your project has two applications named `app, app2` and you want to update to the commit `d42963d007e55cc37ef666019428b1d06a25cf71`, run: +```bash +curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- app,app2 d42963d007e55cc37ef666019428b1d06a25cf71 commit +``` + +If your project has three applications named `foo,bar,baz` and you want to update to the `v.0.8.0` release tag, run: +```bash +curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- foo,bar,baz v0.8.0 tag +``` From e4aadc5bd976439d066d2cd31ebdd9e94b24051e Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 16:29:34 -0700 Subject: [PATCH 78/89] Use relative links --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index fd904ce32..7c556e8f4 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ Applications must meet [these requirements](/template-only-docs/application-requ ### Multiple Applications -You can use this template with multiple applications. By default, this template assumes your project has one application named `app`. However, it's straightforward to [add additional applications](https://github.com/navapbc/template-infra/tree/main/template-only-docs/multiple-applications.md). +You can use this template with multiple applications. By default, this template assumes your project has one application named `app`. However, it's straightforward to [add additional applications](/template-only-docs/multiple-applications.md). ## Installation @@ -34,7 +34,7 @@ To install this template to your project, run the following command in your proj curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/download-and-install-template.sh | bash -s ``` -The [download and install script](https://github.com/navapbc/template-infra/tree/main/template-only-bin/download-and-install-template.sh) clones this template repository, copies the template files to your repository, and removes files that are only relevant to the template itself. +The [download and install script](/template-only-bin/download-and-install-template.sh) clones this template repository, copies the template files to your repository, and removes files that are only relevant to the template itself. Now you're ready to set up the various pieces of your infrastructure. From 1c263f6f506d4f9f7439f68e9737902bb0b87099 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 16:30:25 -0700 Subject: [PATCH 79/89] Fix prereq --- template-only-docs/multiple-applications.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/template-only-docs/multiple-applications.md b/template-only-docs/multiple-applications.md index e6ff6948d..b1ca117ff 100644 --- a/template-only-docs/multiple-applications.md +++ b/template-only-docs/multiple-applications.md @@ -4,7 +4,7 @@ You can use this template with multiple applications. By default, this template ## Prerequisites -* None +* You have [installed](/README.md#installation) this template. ## Instructions to add an additional application From 8310979bf84252f209a42463769b7349f6decf95 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 16:32:20 -0700 Subject: [PATCH 80/89] Add stronger warnings --- template-only-docs/multiple-applications.md | 2 +- template-only-docs/update-template.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/template-only-docs/multiple-applications.md b/template-only-docs/multiple-applications.md index b1ca117ff..ad3da1b07 100644 --- a/template-only-docs/multiple-applications.md +++ b/template-only-docs/multiple-applications.md @@ -16,7 +16,7 @@ Applications must meet [these requirements](/template-only-docs/application-requ Add the application's source code to a folder in the project's root folder (e.g. `/second-app`). -⚠️ Warning: In general, it's best to use short, descriptive one-word names because some AWS resources have character limits. If you must use multiple words, use hyphens (not underscores) to separate each word. +⚠️ **Warning:** In general, it's best to use short, descriptive one-word names because some AWS resources have character limits. If you must use multiple words, use hyphens (not underscores) to separate each word. ### 3. Add infrastructure support for the application diff --git a/template-only-docs/update-template.md b/template-only-docs/update-template.md index f0593baf4..1acdfbd36 100644 --- a/template-only-docs/update-template.md +++ b/template-only-docs/update-template.md @@ -14,7 +14,7 @@ curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only `` is a required argument. It must be a comma-separated list (no spaces) of the apps in `/infra`. App names must be hyphen-separated (i.e. kebab-case). Examples: `app`, `app,app2`, `my-app,your-app`. -**Remember:** Read the release notes in case there are breaking changes you need to address. +⚠️ **Warning:** Read the release notes in case there are breaking changes you need to address manually. ### Specifying a branch, a commit, or a tag From 754621d5aed79cd418c3a8ed6dea157e3a11b5ae Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 16:33:00 -0700 Subject: [PATCH 81/89] Add link to release notes --- template-only-docs/update-template.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/template-only-docs/update-template.md b/template-only-docs/update-template.md index 1acdfbd36..e1f0743b4 100644 --- a/template-only-docs/update-template.md +++ b/template-only-docs/update-template.md @@ -14,7 +14,7 @@ curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only `` is a required argument. It must be a comma-separated list (no spaces) of the apps in `/infra`. App names must be hyphen-separated (i.e. kebab-case). Examples: `app`, `app,app2`, `my-app,your-app`. -⚠️ **Warning:** Read the release notes in case there are breaking changes you need to address manually. +⚠️ **Warning:** Read the [release notes](https://github.com/navapbc/template-infra/releases) in case there are breaking changes you need to address manually. ### Specifying a branch, a commit, or a tag From a6691b270a52c88cd196794036234ed7abb0579d Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 16:38:06 -0700 Subject: [PATCH 82/89] Add missing documentation --- template-only-bin/install-app.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/template-only-bin/install-app.sh b/template-only-bin/install-app.sh index 18d027fc9..12a4e08e6 100644 --- a/template-only-bin/install-app.sh +++ b/template-only-bin/install-app.sh @@ -1,10 +1,13 @@ #!/usr/bin/env bash # ----------------------------------------------------------------------------- -# This script downloads and installs infrastructure for an application. -# Run this script in your project's root directory. +# This script installs infrastructure for an application. +# It is called by other scripts. # # Positional parameters: # APP_NAME (required) - the name for the application, use kebab-case +# DST_PREFIX (optional) - the directory that the application should be installed in +# Defaults to "" (the current directory). If directory is supplied, must contain a +# trailing slash. # ----------------------------------------------------------------------------- set -euo pipefail From c83663418644ff6088bd729f1f78101bdc5e86d4 Mon Sep 17 00:00:00 2001 From: Rocket Date: Tue, 7 May 2024 23:48:22 -0700 Subject: [PATCH 83/89] Consistency --- README.md | 2 +- docs/infra/destroy-infrastructure.md | 6 +++--- docs/infra/making-infra-changes.md | 2 +- docs/infra/set-up-app-build-repository.md | 2 +- docs/infra/set-up-app-config.md | 2 +- docs/infra/set-up-app-database.md | 4 ++-- docs/infra/set-up-app-monitoring-alerts.md | 4 ++-- docs/infra/set-up-app-service.md | 6 +++--- docs/infra/set-up-aws-account.md | 6 +++--- docs/infra/set-up-network-custom-domains.md | 6 +++--- docs/infra/set-up-network-https.md | 6 +++--- template-only-docs/application-requirements.md | 6 +++--- template-only-docs/set-up-ci.md | 12 +++++------- template-only-docs/template-development-workflow.md | 6 +++--- 14 files changed, 34 insertions(+), 36 deletions(-) diff --git a/README.md b/README.md index 7c556e8f4..dae7ba17b 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ This template includes: * **Team workflows** - templates for pull requests (PRs), architecture decision records (ADRs), and Makefiles * **Account level foundational infrastructure** - infrastructure for Terraform backends, including an S3 bucket and DynamoDB table for storing and managing Terraform state files * **Application infrastructure** - the infrastructure you need to set up a basic web app, including container image repository, load balancer, web service, and database -* **Continuous integration (CI) for infrastructure** - GitHub action that performs infra code checks, including linting, validation, and security compliance checks +* **Continuous integration (CI) for infrastructure** - GitHub action that performs infrastructre code checks, including linting, validation, and security compliance checks * **Continous deployment (CD)** - infrastructure for continuous deployment, including AWS account access for Github actions, scripts for building and publishing release artifacts, and a Github action for automated deployments from the main branch * **Documentation** - technical documentation for the decisions that went into all the defaults that come with the template diff --git a/docs/infra/destroy-infrastructure.md b/docs/infra/destroy-infrastructure.md index 0635e3302..874f229b4 100644 --- a/docs/infra/destroy-infrastructure.md +++ b/docs/infra/destroy-infrastructure.md @@ -11,7 +11,7 @@ To destroy everything you'll need to undeploy all the infrastructure in reverse $ terraform destroy -var-file=dev.tfvars ``` -2. Then to destroy the backends, first you'll need to add `force_destroy = true` to the S3 buckets, and update the lifecycle block to set `prevent_destroy = false`. Then run `terraform apply` from within the `infra/accounts` directory. The reason we need to do this is because S3 buckets by default are protected from destruction to avoid loss of data. See [Terraform: Destroy/Replace Buckets](https://medium.com/interleap/terraform-destroy-replace-buckets-cf9d63d0029d) for a more in-depth explanation. +2. Then to destroy the backends, first you'll need to add `force_destroy = true` to the S3 buckets, and update the lifecycle block to set `prevent_destroy = false`. Then run `terraform apply` from within the `/infra/accounts` directory. The reason we need to do this is because S3 buckets by default are protected from destruction to avoid loss of data. See [Terraform: Destroy/Replace Buckets](https://medium.com/interleap/terraform-destroy-replace-buckets-cf9d63d0029d) for a more in-depth explanation. ```terraform # infra/modules/modules/terraform-backend-s3/main.tf @@ -46,13 +46,13 @@ To destroy everything you'll need to undeploy all the infrastructure in reverse }2 ``` -4. Then run the following from within the `infra/accounts` directory to copy the `tfstate` back to a local `tfstate` file: +4. Then run the following from within the `/infra/accounts` directory to copy the `tfstate` back to a local `tfstate` file: ```bash terraform init -force-copy ``` -5. Finally, you can run `terraform destroy` within the `infra/accounts` directory. +5. Finally, you can run `terraform destroy` within the `/infra/accounts` directory. ```bash terraform destroy diff --git a/docs/infra/making-infra-changes.md b/docs/infra/making-infra-changes.md index e01d79100..e8f125042 100644 --- a/docs/infra/making-infra-changes.md +++ b/docs/infra/making-infra-changes.md @@ -48,7 +48,7 @@ Look in the script files for more details on usage. ## Using Terraform CLI directly -Finally, if the wrapper scripts don't meet your needs, you can always run `terraform` directly from the root module directory. You may need to do this if you are running Terraform commands other than `terraform plan` and `terraform apply`, such as `terraform import`, `terraform taint`, etc. To do this, you'll need to pass in the appropriate `tfvars` and `tfbackend` files to `terraform init` and `terraform apply`. For example, to make changes to the application's service resources in the dev environment, change to the `infra/app/service` directory and run: +Finally, if the wrapper scripts don't meet your needs, you can always run `terraform` directly from the root module directory. You may need to do this if you are running Terraform commands other than `terraform plan` and `terraform apply`, such as `terraform import`, `terraform taint`, etc. To do this, you'll need to pass in the appropriate `.tfbackend` files to `terraform init` and `terraform apply`. For example, to make changes to the application's service resources in the dev environment, change to the `/infra/app/service` directory and run: ```bash infra/app/service$ terraform init -backend-config=dev.s3.tfbackend diff --git a/docs/infra/set-up-app-build-repository.md b/docs/infra/set-up-app-build-repository.md index 3a5dcca05..61fc1908a 100644 --- a/docs/infra/set-up-app-build-repository.md +++ b/docs/infra/set-up-app-build-repository.md @@ -34,7 +34,7 @@ To create the `.tfbackend` file for the build repository, run: make infra-configure-app-build-repository APP_NAME= ``` -`APP_NAME` should be the name of the application folder within the `infra` folder. +`` must be the name of the application folder within the `/infra` folder. ### 3. Create build repository resources diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md index 9668d6184..c93413d42 100644 --- a/docs/infra/set-up-app-config.md +++ b/docs/infra/set-up-app-config.md @@ -27,7 +27,7 @@ To use [feature flags](/docs/feature-flags.md), modify the values in the applica ### 3. Configure each environment -Within the application's `app-config` directory (e.g. `infra//app-config`), each environment configured in the `environments` array in the previous step needs its own config file. For example, if the application has three environments `dev`, `staging`, and `prod`, there must be corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. +Within the application's `app-config` directory (e.g. `/infra//app-config`), each environment configured in the `environments` array in the previous step needs its own config file. For example, if the application has three environments `dev`, `staging`, and `prod`, there must be corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. In each environment config file, modify the following values: diff --git a/docs/infra/set-up-app-database.md b/docs/infra/set-up-app-database.md index 7012cf79b..628602af4 100644 --- a/docs/infra/set-up-app-database.md +++ b/docs/infra/set-up-app-database.md @@ -41,9 +41,9 @@ To create the `.tfbackend` file for the new application environment, run: make infra-configure-app-database APP_NAME= ENVIRONMENT= ``` -`APP_NAME` should be the name of the application folder within the `infra` folder. +`` must be the name of the application folder within the `/infra` folder. -`ENVIRONMENT` should be the name of the environment to update. This will create a file called `.s3.tfbackend` in `infra//database`. +`` must be the name of the environment to update. This will create a file called `.s3.tfbackend` in `/infra//database`. ### 3. Create database resources diff --git a/docs/infra/set-up-app-monitoring-alerts.md b/docs/infra/set-up-app-monitoring-alerts.md index 5c419bf32..c021acf9b 100644 --- a/docs/infra/set-up-app-monitoring-alerts.md +++ b/docs/infra/set-up-app-monitoring-alerts.md @@ -37,9 +37,9 @@ To apply the changes, run the following command. Review the Terraform output car make infra-update-app-service APP_NAME= ENVIRONMENT= ``` -`APP_NAME` must be the name of the application folder within the `infra` folder. +`` must be the name of the application folder within the `/infra` folder. -`ENVIRONMENT` must be the name of the environment to update. +`` must be the name of the environment to update. ## Instructions to set up external incident management service integration diff --git a/docs/infra/set-up-app-service.md b/docs/infra/set-up-app-service.md index 9d6b2674f..665a5a911 100644 --- a/docs/infra/set-up-app-service.md +++ b/docs/infra/set-up-app-service.md @@ -36,15 +36,15 @@ aws iam list-account-aliases ### 2. Configure backend -To create the `.tfbackend` and `.tfvars` files for the new application service, run: +To create the `.tfbackend` files for the new application service, run: ```bash make infra-configure-app-service APP_NAME= ENVIRONMENT= ``` -`APP_NAME` must be the name of the application folder within the `infra` folder. +`` must be the name of the application folder within the `/infra` folder. -`ENVIRONMENT` must be the name of the environment to update. This will create a file called `.s3.tfbackend` in `infra//service`. +`` must be the name of the environment to update. This will create a file called `.s3.tfbackend` in `/infra//service`. ### 3. Build and publish the application to the build repository diff --git a/docs/infra/set-up-aws-account.md b/docs/infra/set-up-aws-account.md index aab6275e0..b0831df0d 100644 --- a/docs/infra/set-up-aws-account.md +++ b/docs/infra/set-up-aws-account.md @@ -30,15 +30,15 @@ To see a more human readable account alias instead of the account, run: aws iam list-account-aliases ``` -### 2. Create backend resources and tfbackend config file +### 2. Create backend resources and `.tfbackend` config file -Run the following command, replacing `` with a human readable name for the AWS account that you're authenticated into. The account name will be used to prefix the tfbackend file. For example, if you have an account per environment, the account name can be the name of the environment (e.g. "prod" or "staging"). Or if you are setting up an account for all lower environments, the account name can be "lowers". If your AWS account has an account alias, you can also use that. +Run the following command, replacing `` with a human readable name for the AWS account that you're authenticated into. The account name will be used to prefix the `.tfbackend` file. For example, if you have an account per environment, the account name can be the name of the environment (e.g. "prod" or "staging"). Or if you are setting up an account for all lower environments, the account name can be "lowers". If your AWS account has an account alias, you can also use that. ```bash make infra-set-up-account ACCOUNT_NAME= ``` -This command will create the S3 tfstate bucket and the GitHub OIDC provider. It will also create a `[account name].[account id].s3.tfbackend` file in the `infra/accounts` directory. +This command will create the S3 tfstate bucket and the GitHub OIDC provider. It will also create a `[account name].[account id].s3.tfbackend` file in the `/infra/accounts` directory. ## Making changes to an AWS account diff --git a/docs/infra/set-up-network-custom-domains.md b/docs/infra/set-up-network-custom-domains.md index 0fbc071a5..289fa7cff 100644 --- a/docs/infra/set-up-network-custom-domains.md +++ b/docs/infra/set-up-network-custom-domains.md @@ -87,7 +87,7 @@ nslookup -type=NS **For each application** in the network that should use the custom domain, perform the following. -Within the `app-config` directory (e.g. `infra//app-config`), each environment has its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. +Within the `app-config` directory (e.g. `/infra//app-config`), each environment has its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. In each environment config file, define the `domain_name`. @@ -101,9 +101,9 @@ The `domain_name` must be either the same as the `hosted_zone` or a subdomain of make infra-update-app-service APP_NAME= ENVIRONMENT= ``` -`APP_NAME` must be the name of the application folder within the `infra` folder. +`` must be the name of the application folder within the `/infra` folder. -`ENVIRONMENT` must be the name of the environment to update. +`` must be the name of the environment to update. ## If a network does not need custom domains diff --git a/docs/infra/set-up-network-https.md b/docs/infra/set-up-network-https.md index 95dfefff8..2fd6536c1 100644 --- a/docs/infra/set-up-network-https.md +++ b/docs/infra/set-up-network-https.md @@ -55,7 +55,7 @@ aws acm describe-certificate --certificate-arn --query Certifi **For each application and environment** that should use HTTPS, perform the following. -Within the `app-config` directory (e.g. `infra//app-config`), each environment has its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. +Within the `app-config` directory (e.g. `/infra//app-config`), each environment has its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. In each environment config file, set `enable_https` to `true`. This will attach the SSL/TLS certificate to the load balancer. @@ -69,9 +69,9 @@ You should have already set `domain_name` as part of [setting up custom domain n make infra-update-app-service APP_NAME= ENVIRONMENT= ``` -`APP_NAME` must be the name of the application folder within the `infra` folder. +`` must be the name of the application folder within the `/infra` folder. -`ENVIRONMENT` must be the name of the environment to update. +`` must be the name of the environment to update. ## If a network does not need HTTPS diff --git a/template-only-docs/application-requirements.md b/template-only-docs/application-requirements.md index d3caace1a..8637b451d 100644 --- a/template-only-docs/application-requirements.md +++ b/template-only-docs/application-requirements.md @@ -22,13 +22,13 @@ If your application needs a database, it must also: ## Example Application -The infra template includes an example "hello, world" application that works with the template. The source code for this test application is at [app](/app). +This template includes an example "hello, world" application that works with the template. The source code for this test application is at [`/app`](/app). -A live demo of the test application is fully deployed by the repo, which is used for testing the infra template. Please check [that repo's README](https://github.com/navapbc/platform-test?tab=readme-ov-file#environment-urls) to locate a URL for seeing the live demo. +A live demo of the test application is fully deployed by the repo, which is used for testing this template. Please check [that repo's README](https://github.com/navapbc/platform-test?tab=readme-ov-file#environment-urls) to locate a URL for seeing the live demo. ## Template Applications -You can use the following template applications with the template infrastructure. Each of these includes a script to generate a working application that works with this infra template. +You can use the following template applications with this infrastructure template. Each of these includes a script to generate a working application that works with this template. * [template-application-nextjs](https://github.com/navapbc/template-application-nextjs) * [template-application-flask](https://github.com/navapbc/template-application-flask) diff --git a/template-only-docs/set-up-ci.md b/template-only-docs/set-up-ci.md index 09b2f6504..f8eccc13a 100644 --- a/template-only-docs/set-up-ci.md +++ b/template-only-docs/set-up-ci.md @@ -4,12 +4,10 @@ CI should automatically be set up once the CI files in `.github/workflows` are committed and pushed to the remote repository in GitHub. -Some checks are disabled until you've completed certain setup steps: +### Instructions -### After completing infra setup +Some checks are disabled until you've completed all set up steps: -After completing the [infra setup](/infra/README.md#instructions): - -* Uncomment the infra end-to-end tests by searching for `!!` in [ci-infra-service.yml](/.github/workflows/ci-infra-service.yml). You can verify that CI is running and passing by clicking into the Actions tab in GitHub. - * Note that this repo only contains CI for infra. If you're using one of the [Platform application templates](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), then the application CI (`/.github/workflows/ci-app.yml`) is already included. Otherwise, you'll need to create one. -* If you setup your AWS account in a different region than `us-east-1`, update the `aws-region` workflow settings in [`/.github/workflows/check-infra-auth.yml`](/.github/workflows/check-infra-auth.yml) to match your region. +1. Uncomment the infrastructure end-to-end tests by searching for `!!` in [`.github/workflows/ci-infra-service.yml`](/.github/workflows/ci-infra-service.yml). After uncommenting, verify that the CI is running and passing by clicking the Actions tab in GitHub. + * Note that this repo only contains CI for the infrastracture. If you're using one of the [Platform application templates](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), then the application CI workflow (`/.github/workflows/ci-app.yml`) is already included. Otherwise, you'll need to create one. +2. If you setup your AWS account in a region other than `us-east-1`, update the `aws-region` workflow settings in [`/.github/workflows/check-infra-auth.yml`](/.github/workflows/check-infra-auth.yml) to match your region. diff --git a/template-only-docs/template-development-workflow.md b/template-only-docs/template-development-workflow.md index a5ddbbc01..ef80b81ae 100644 --- a/template-only-docs/template-development-workflow.md +++ b/template-only-docs/template-development-workflow.md @@ -4,7 +4,7 @@ This is the workflow for developers making changes to the infrastructure templat ## Prerequisites -For most infrastructure changes, you will need an environment to work with. Since template-infra is a template and not a live project, it doesn't have any long-lived environments. Thus, you should develop and test your infrastructure changes using the `dev` environment on one of the following test repos: +For most infrastructure changes, you will need an environment to work with. Since this repo is a template and not a live project, it doesn't have any long-lived environments. Thus, you should develop and test your infrastructure changes using the `dev` environment on one of the following test repos: * [platform-test](https://github.com/navapbc/platform-test) – Test project that uses [template-infra](https://github.com/navapbc/template-infra) and the [example app](https://github.com/navapbc/template-infra/tree/main/app) that comes with the template. This is the default project we use for development and testing infrastructure changes. * [platform-test-flask](https://github.com/navapbc/platform-test-flask) - Test project that uses [template-infra](https://github.com/navapbc/template-infra) and [template-application-flask](https://github.com/navapbc/template-application-flask) @@ -31,7 +31,7 @@ On the platform test repo, you'll do the following: 4. Create a pull request 5. Iterate until all CI checks pass on your PR and you’ve also done additional testing that you need to validate. *Do not merge the PR.* -### 2. Create a pull request on infra template repo +### 2. Create a pull request on the `template-infra` repo 1. Once you've completed development and testing, create a pull request on the [template-infra](https://github.com/navapbc/template-infra) repo with the same changes you made on the platform test repo. 2. In the "Testing" section of the PR description, link to the PR on the platform test repo as evidence of the testing you did to verify your changes. @@ -39,7 +39,7 @@ On the platform test repo, you'll do the following: ### 3. Push changes to platform test repos -In most cases, after you merge changes to the infra template, the changes will be automatically pushed to the various platform test repos. However, the following changes aren't automatically propagated to the platform test repos: +In most cases, after you merge changes to the `template-infra` repo, the changes will be automatically pushed to the various platform test repos. However, the following changes aren't automatically propagated to the platform test repos: 1. Changes to [this list of files in update-template.sh](https://github.com/navapbc/template-infra/blob/main/template-only-bin/update-template.sh#L17-L28) 2. Deletions of template files From 5fd5b3b491774578b9a2405461af6631d5feb7dc Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 16 May 2024 10:38:46 -0700 Subject: [PATCH 84/89] Address shellcheck --- template-only-bin/download-and-install-app.sh | 2 +- template-only-bin/install-app.sh | 6 ++- template-only-bin/update-template.sh | 47 ++++++++++--------- 3 files changed, 31 insertions(+), 24 deletions(-) diff --git a/template-only-bin/download-and-install-app.sh b/template-only-bin/download-and-install-app.sh index 680934e09..95c927020 100755 --- a/template-only-bin/download-and-install-app.sh +++ b/template-only-bin/download-and-install-app.sh @@ -12,7 +12,7 @@ APP_NAME=$1 CURRENT_VERSION=$(cat .template-version) # Enforce kebab-case -APP_NAME_KEBAB=$(echo $APP_NAME | tr "_" "-") +APP_NAME_KEBAB=$(echo "$APP_NAME" | tr "_" "-") echo "Cloning template-infra..." git clone https://github.com/navapbc/template-infra.git diff --git a/template-only-bin/install-app.sh b/template-only-bin/install-app.sh index 12a4e08e6..ffc1939e1 100644 --- a/template-only-bin/install-app.sh +++ b/template-only-bin/install-app.sh @@ -20,7 +20,11 @@ cp -r template-infra/infra/app "${DST_PREFIX}infra/$APP_NAME" # Helper to get the correct sed -i behavior for both GNU sed and BSD sed (installed by default on macOS) # Hat tip: https://stackoverflow.com/a/38595160 sedi () { - sed --version >/dev/null 2>&1 && sed -i -- "$@" || sed -i "" "$@" + if sed --version >/dev/null 2>&1; then + sed -i -- "$@" + else + sed -i "" "$@" + fi } # Export the function so it can be used below export -f sedi diff --git a/template-only-bin/update-template.sh b/template-only-bin/update-template.sh index fdb78f750..06cfcc562 100755 --- a/template-only-bin/update-template.sh +++ b/template-only-bin/update-template.sh @@ -47,6 +47,7 @@ echo "=====================================================================" echo "Updating template-infra" echo "=====================================================================" echo "APP_NAMES=$APP_NAMES" +echo "CURRENT_VERSION=$CURRENT_VERSION" echo "TARGET_VERSION=$TARGET_VERSION" echo "TARGET_VERSION_TYPE=$TARGET_VERSION_TYPE" echo @@ -102,50 +103,52 @@ git fetch upstream-template-infra >& /dev/null echo "Converting $TARGET_VERSION to hash..." case $TARGET_VERSION_TYPE in "branch") - TARGET_VERSION_HASH="$(git rev-parse upstream-template-infra/$TARGET_VERSION)" + TARGET_VERSION_HASH=$(git rev-parse "upstream-template-infra/$TARGET_VERSION") ;; "commit") echo "No conversion needed." TARGET_VERSION_HASH=$TARGET_VERSION ;; "tag") - TARGET_VERSION_HASH="$(git ls-remote --tags upstream-template-infra $TARGET_VERSION | cut -d$'\t' -f1)" + TARGET_VERSION_HASH=$(git ls-remote --tags upstream-template-infra "$TARGET_VERSION" | cut -d$'\t' -f1) ;; esac echo "TARGET_VERSION_HASH=$TARGET_VERSION_HASH" echo # Note: Keep this list in sync with the files copied in install-template.sh -INCLUDE_PATHS=" - .github - bin - docs - infra - Makefile - .dockleconfig - .grype.yml - .hadolint.yaml - .trivyignore" +INCLUDE_PATHS=( + ".github" + "bin" + "docs" + "infra" + "Makefile" + ".dockleconfig" + ".grype.yml" + ".hadolint.yaml" + ".trivyignore" +) # Note: Exclude terraform deployment files, and CI/CD workflows as those are handled # separately below. -EXCLUDE_PATHS=" - ':!*.terraform*' - ':!*.tfbackend' - ':!.github/workflows/'" +EXCLUDE_PATHS=( + "':!*.terraform*'" + "':!*.tfbackend'" + "':!.github/workflows/'" +) # Exclude all applications for APP_NAME in ${APP_NAMES//,/ } do - EXCLUDE_PATHS="${EXCLUDE_PATHS} ':!infra/$APP_NAME'" + EXCLUDE_PATHS+=("':!infra/$APP_NAME'") done # Show the changes to be made -STAT_COMMAND="git --no-pager diff -R --stat $TARGET_VERSION_HASH -- $(echo $INCLUDE_PATHS) $(echo $EXCLUDE_PATHS)" +STAT_COMMAND="git --no-pager diff -R --stat $TARGET_VERSION_HASH -- ${INCLUDE_PATHS[*]} ${EXCLUDE_PATHS[*]}" eval "$STAT_COMMAND" # Make the patch file -DIFF_COMMAND="git diff -R $TARGET_VERSION_HASH -- $(echo $INCLUDE_PATHS) $(echo $EXCLUDE_PATHS)" +DIFF_COMMAND="git diff -R $TARGET_VERSION_HASH -- ${INCLUDE_PATHS[*]} ${EXCLUDE_PATHS[*]}" eval "$DIFF_COMMAND > main-template.patch" # Apply the patch file @@ -170,7 +173,7 @@ do echo "---------------------------------------------------------------------" # If the APP_NAME is not named `app`, then install a new app in template-infra to diff against if [ "$APP_NAME" != "app" ]; then - curl "https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/install-app.sh" | bash -s -- $APP_NAME "template-infra/" + curl "https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/install-app.sh" | bash -s -- "$APP_NAME" "template-infra/" echo fi @@ -184,11 +187,11 @@ do git diff --no-index --dst-prefix="" "infra/$APP_NAME" "template-infra/infra/$APP_NAME" > "template-infra/$APP_NAME.patch" || true # The stat version of the `git-apply` command`, used to output the changes to STDOUT - STAT_COMMAND="git --no-pager apply --stat --allow-empty template-infra/$APP_NAME.patch --exclude='*.tfbackend' --exclude='*.terraform*'" + STAT_COMMAND="git --no-pager apply --stat --allow-empty template-infra/$APP_NAME.patch --exclude='*.tfbackend' --exclude='*.terraform*' --exclude='*.tfstate*'" eval "$STAT_COMMAND" # Actually run the `git apply` command - git apply --allow-empty "template-infra/$APP_NAME.patch" --exclude="*.tfbackend" --exclude="*.terraform*" + git apply --allow-empty "template-infra/$APP_NAME.patch" --exclude="*.tfbackend" --exclude="*.terraform*" --exclude="*.tfstate*" # Increment step counter STEP_COUNT=$((STEP_COUNT+1)) From 63d57c1f237ca7e6997d441871732faf5115fc23 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 16 May 2024 10:39:16 -0700 Subject: [PATCH 85/89] Alternate app update to avoid no-index where possible --- template-only-bin/update-template.sh | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/template-only-bin/update-template.sh b/template-only-bin/update-template.sh index 06cfcc562..e783eae52 100755 --- a/template-only-bin/update-template.sh +++ b/template-only-bin/update-template.sh @@ -177,14 +177,20 @@ do echo fi + # Copy the latest files into the working tree + cp -r "template-infra/infra/$APP_NAME" infra + + # Add any untracked files as intent-to-add + git add --intent-to-add "infra/$APP_NAME" + # To create a git patch comparing a project's application: - # --no-index allows us comparison between differently-named or -nested directories - # --dst-prefix="" removes the destination prefix, essentially having the effect of - # removing the additional `template-infra/` path prefix, making the patch appliable # || true is necessary because this bash script includes `set -e` option, which will # immediately exit for any non-zero exit codes. That's generally correct, but # `git diff --no-index` will return 1 to indicate differences between the files. - git diff --no-index --dst-prefix="" "infra/$APP_NAME" "template-infra/infra/$APP_NAME" > "template-infra/$APP_NAME.patch" || true + git diff "infra/$APP_NAME" > "template-infra/$APP_NAME.patch" || true + + # Restore infra dir + git checkout "infra/$APP_NAME" # The stat version of the `git-apply` command`, used to output the changes to STDOUT STAT_COMMAND="git --no-pager apply --stat --allow-empty template-infra/$APP_NAME.patch --exclude='*.tfbackend' --exclude='*.terraform*' --exclude='*.tfstate*'" From 2d6c3e7bbe9cc3c57fa5638c2b423b3239b5e7e0 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 16 May 2024 11:34:46 -0700 Subject: [PATCH 86/89] Address shellcheck --- template-only-bin/download-and-install-app.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/template-only-bin/download-and-install-app.sh b/template-only-bin/download-and-install-app.sh index 95c927020..774db5a42 100755 --- a/template-only-bin/download-and-install-app.sh +++ b/template-only-bin/download-and-install-app.sh @@ -23,7 +23,7 @@ git checkout "$CURRENT_VERSION" >& /dev/null cd - >& /dev/null # Install the app -curl "https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/install-app.sh" | bash -s -- $APP_NAME +curl "https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/install-app.sh" | bash -s -- "$APP_NAME_KEBAB" echo "Cleaning up template-infra folder..." rm -fr template-infra From 9003c8473076b76784f4d72da3da7d19203f6966 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 16 May 2024 11:41:54 -0700 Subject: [PATCH 87/89] Update template-only CI/CD --- .github/workflows/template-only-cd.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/template-only-cd.yml b/.github/workflows/template-only-cd.yml index bf9a7c10d..5d7935ba7 100644 --- a/.github/workflows/template-only-cd.yml +++ b/.github/workflows/template-only-cd.yml @@ -34,7 +34,7 @@ jobs: - name: Update infra template working-directory: project-repo - run: ../template-infra/template-only-bin/update-template.sh + run: ../template-infra/template-only-bin/update-template.sh app - name: Push changes to project repo working-directory: project-repo From 8ec917a45b8bc2917342feb77f339e0cffb588dd Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 16 May 2024 13:45:32 -0700 Subject: [PATCH 88/89] Move auth with AWS into prereqs instead of step 1 --- docs/infra/set-up-app-build-repository.md | 19 +++------------ docs/infra/set-up-app-database.md | 23 ++++-------------- docs/infra/set-up-app-service.md | 21 ++++------------- docs/infra/set-up-aws-account.md | 17 ++------------ docs/infra/set-up-infrastructure-tools.md | 5 ++++ docs/infra/set-up-network-custom-domains.md | 25 +++++--------------- docs/infra/set-up-network-https.md | 26 +++++---------------- docs/infra/set-up-network.md | 19 +++------------ 8 files changed, 34 insertions(+), 121 deletions(-) diff --git a/docs/infra/set-up-app-build-repository.md b/docs/infra/set-up-app-build-repository.md index 61fc1908a..9652fe773 100644 --- a/docs/infra/set-up-app-build-repository.md +++ b/docs/infra/set-up-app-build-repository.md @@ -6,27 +6,14 @@ The application build repository setup process will create infrastructure resour ## Prerequisites +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. * You have [set up the AWS account(s)](./set-up-aws-accounts.md). * You have [configured the application](/infra/app/app-config/main.tf). * You have [set up the network(s)](./set-up-networks.md). ## Instructions -### 1. Make sure you're authenticated into the AWS account where you want to deploy resources shared across environments - -This setup applies to the account you're authenticated into. To see which account that is, run: - -```bash -aws sts get-caller-identity -``` - -To see a more human readable account alias instead of the account, run: - -```bash -aws iam list-account-aliases -``` - -### 2. Configure backend +### 1. Configure backend To create the `.tfbackend` file for the build repository, run: @@ -36,7 +23,7 @@ make infra-configure-app-build-repository APP_NAME= `` must be the name of the application folder within the `/infra` folder. -### 3. Create build repository resources +### 2. Create build repository resources To create the resources, run the following command. Review the Terraform output carefully before typing "yes" to apply the changes. diff --git a/docs/infra/set-up-app-database.md b/docs/infra/set-up-app-database.md index 628602af4..fb9567b6d 100644 --- a/docs/infra/set-up-app-database.md +++ b/docs/infra/set-up-app-database.md @@ -12,6 +12,7 @@ The database set up process will: ## Prerequisites +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. * You have [set up the AWS account(s)](./set-up-aws-accounts.md). * You have [configured the application](/infra/app/app-config/main.tf). * You have [set up the network(s)](./set-up-networks.md). @@ -19,21 +20,7 @@ The database set up process will: ## Instructions -### 1. Make sure you're authenticated into the AWS account where you want to deploy this environment - -This setup applies to the account you're authenticated into. To see which account that is, run: - -```bash -aws sts get-caller-identity -``` - -To see a more human readable account alias instead of the account, run: - -```bash -aws iam list-account-aliases -``` - -### 2. Configure backend +### 1. Configure backend To create the `.tfbackend` file for the new application environment, run: @@ -45,7 +32,7 @@ make infra-configure-app-database APP_NAME= ENVIRONMENT= `` must be the name of the environment to update. This will create a file called `.s3.tfbackend` in `/infra//database`. -### 3. Create database resources +### 2. Create database resources To create the resources, run the following command. Review the Terraform output carefully before typing "yes" to apply the changes. This can take over 5 minutes. @@ -53,7 +40,7 @@ To create the resources, run the following command. Review the Terraform output make infra-update-app-database APP_NAME= ENVIRONMENT= ``` -### 4. Create Postgres users +### 3. Create Postgres users To trigger the role manager Lambda function that was created in the previous step to create the `app` and `migrator` Postgres users, run: @@ -96,7 +83,7 @@ This will cause all future tables created by the `migrator` user to automaticall Why is this needed? The reason is that the `migrator` role will be used by the migration task to run database migrations (creating tables, altering tables, etc.), while the `app` role will be used by the web service to access the database. Moreover, in Postgres, new tables won't automatically be accessible by roles other than the creator unless specifically granted, even if those other roles have usage access to the schema that the tables are created in. In other words, if the `migrator` user created a new table `foo` in the `app` schema, the `app` user will not automatically be able to access it by default. -### 5. Check that database roles have been configured properly +### 4. Check that database roles have been configured properly Verify the that the database roles have been configured properly by running: diff --git a/docs/infra/set-up-app-service.md b/docs/infra/set-up-app-service.md index 665a5a911..8aaeacb07 100644 --- a/docs/infra/set-up-app-service.md +++ b/docs/infra/set-up-app-service.md @@ -12,6 +12,7 @@ The application service set up process will: ## Prerequisites +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. * You have [set up the AWS account(s)](./set-up-aws-accounts.md). * You have [configured the application](/infra/app/app-config/main.tf). * You have [set up the network(s)](./set-up-networks.md). @@ -20,21 +21,7 @@ The application service set up process will: ## Instructions -### 1. Make sure you're authenticated into the AWS account where you want to deploy this environment - -This setup applies to the account you're authenticated into. To see which account that is, run: - -```bash -aws sts get-caller-identity -``` - -To see a more human readable account alias instead of the account, run: - -```bash -aws iam list-account-aliases -``` - -### 2. Configure backend +### 1. Configure backend To create the `.tfbackend` files for the new application service, run: @@ -46,7 +33,7 @@ make infra-configure-app-service APP_NAME= ENVIRONMENT= `` must be the name of the environment to update. This will create a file called `.s3.tfbackend` in `/infra//service`. -### 3. Build and publish the application to the build repository +### 2. Build and publish the application to the build repository Before creating the application resources, you need to build and publish at least one image to the build repository. This step does not need to be run per environment. @@ -59,7 +46,7 @@ make release-publish APP_NAME= Copy the image tag name that is output. You'll need this in the next step. -### 4. Create application resources with the image tag that was published +### 3. Create application resources with the image tag that was published To create the resources, run the following command using the image tag output by the previous step. Review the Terraform output carefully before typing "yes" to apply the changes. This can take over 5 minutes. diff --git a/docs/infra/set-up-aws-account.md b/docs/infra/set-up-aws-account.md index b0831df0d..2850fb0aa 100644 --- a/docs/infra/set-up-aws-account.md +++ b/docs/infra/set-up-aws-account.md @@ -10,27 +10,14 @@ The AWS account setup process will: ## Prerequisites +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. * You have [set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md). * You have [configured the project](/infra/project-config/main.tf). * You have [decided on your environment and AWS account strategy](./set-up-aws-accounts.md). ## Instructions -### 1. Make sure you're authenticated into the AWS account you want to configure - -This setup applies to the account you're authenticated into. To see which account that is, run: - -```bash -aws sts get-caller-identity -``` - -To see a more human readable account alias instead of the account, run: - -```bash -aws iam list-account-aliases -``` - -### 2. Create backend resources and `.tfbackend` config file +### 1. Create backend resources and `.tfbackend` config file Run the following command, replacing `` with a human readable name for the AWS account that you're authenticated into. The account name will be used to prefix the `.tfbackend` file. For example, if you have an account per environment, the account name can be the name of the environment (e.g. "prod" or "staging"). Or if you are setting up an account for all lower environments, the account name can be "lowers". If your AWS account has an account alias, you can also use that. diff --git a/docs/infra/set-up-infrastructure-tools.md b/docs/infra/set-up-infrastructure-tools.md index a16c8745b..7cb9b51bf 100644 --- a/docs/infra/set-up-infrastructure-tools.md +++ b/docs/infra/set-up-infrastructure-tools.md @@ -88,6 +88,11 @@ To use Terraform with your AWS accounts, you must configure your AWS credentials ```bash aws sts get-caller-identity ``` + To see a more human-readable account alias instead of the account, run: + ```bash + aws iam list-account-aliases + ``` + ### References diff --git a/docs/infra/set-up-network-custom-domains.md b/docs/infra/set-up-network-custom-domains.md index 289fa7cff..c211b9167 100644 --- a/docs/infra/set-up-network-custom-domains.md +++ b/docs/infra/set-up-network-custom-domains.md @@ -12,6 +12,7 @@ The custom domain set up process will: ## Prerequisites * You have registered custom domain(s) with a domain registrar (e.g. Namecheap, GoDaddy, Google Domains, etc.). +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. * You have [set up the AWS account(s)](./set-up-aws-accounts.md). * You have [configured all application(s)](./set-up-app-config.md). * You have [set up the networks](./set-up-network.md) that you want to add the custom domain to. @@ -19,21 +20,7 @@ The custom domain set up process will: ## Instructions -### 1. Make sure you're authenticated into the AWS account you want to configure - -This setup applies to the account you're authenticated into. To see which account that is, run: - -```bash -aws sts get-caller-identity -``` - -To see a more human readable account alias instead of the account, run: - -```bash -aws iam list-account-aliases -``` - -### 2. Set hosted zone in domain configuration +### 1. Set hosted zone in domain configuration The custom domain configuration is defined as a `domain_config` object in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf). A [hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) represents a domain and all of its subdomains. For example, a hosted zone of `platform-test.navateam.com` includes `platform-test.navateam.com`, `cdn.platform-test.navateam.com`, `notifications.platform-test.navateam.com`, `foo.bar.platform-test.navateam.com`, etc. @@ -42,7 +29,7 @@ The custom domain configuration is defined as a `domain_config` object in [`/inf 1. Set the `hosted_zone` to match the custom domain (or a subdomain of the custom domain) that you registered. 2. Set `manage_dns` to `true`. -### 3. Update the network layer to create the hosted zones +### 2. Update the network layer to create the hosted zones **For each network** you that you added a custom domain to in the previous step, run the following command to create the hosted zone specified in the domain configuration: @@ -50,7 +37,7 @@ The custom domain configuration is defined as a `domain_config` object in [`/inf make infra-update-network NETWORK_NAME= ``` -### 4. Delegate DNS requests to the newly created hosted zone +### 3. Delegate DNS requests to the newly created hosted zone You most likely registered your domain outside of this project. Using whichever service you used to register the domain name (e.g. Namecheap, GoDaddy, Google Domains, etc.), add a DNS NS (nameserver) record. Set the "name" equal to the `hosted_zone` and set the value equal to the list of hosted zone name servers that was created in the previous step. @@ -83,7 +70,7 @@ Verify that DNS requests are being served by the hosted zone nameservers by runn nslookup -type=NS ``` -### 5. Create DNS A (address) records to route traffic from the custom domain to the application's load balancer +### 4. Create DNS A (address) records to route traffic from the custom domain to the application's load balancer **For each application** in the network that should use the custom domain, perform the following. @@ -93,7 +80,7 @@ In each environment config file, define the `domain_name`. The `domain_name` must be either the same as the `hosted_zone` or a subdomain of the `hosted_zone`. For example, if your hosted zone is `platform-test.navateam.com`, then `platform-test.navateam.com` and `cdn.platform-test.navateam.com` are both valid values for `domain_name`. -### 6. Update the application service +### 5. Update the application service **For each application and each environment** in the network that should use the custom domain, apply the changes by running the following command. Review the Terraform output carefully before typing "yes" to apply the changes. diff --git a/docs/infra/set-up-network-https.md b/docs/infra/set-up-network-https.md index 2fd6536c1..f96b649d0 100644 --- a/docs/infra/set-up-network-https.md +++ b/docs/infra/set-up-network-https.md @@ -11,33 +11,19 @@ The HTTPS set up process will: ## Prerequisites +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. * You have [set up custom domains](./set-up-network-custom-domains.md) and met all of those prerequisites. - -This is because SSL/TLS certificates must be properly configured for the specific domain to support establishing secure connections. + * This is because SSL/TLS certificates must be properly configured for the specific domain to support establishing secure connections. ## Instructions -### 1. Make sure you're authenticated into the AWS account you want to configure - -This setup applies to the account you're authenticated into. To see which account that is, run: - -```bash -aws sts get-caller-identity -``` - -To see a more human readable account alias instead of the account, run: - -```bash -aws iam list-account-aliases -``` - -### 2. Set desired certificates in domain configuration +### 1. Set desired certificates in domain configuration **For each network** you want to configure, modify the network in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) to set the `certificate_configs` key. Set the `source` of the domain or subdomain to `issued`. -### 3. Update the network layer to issue the certificates +### 2. Update the network layer to issue the certificates **For each network** you configured in the previous step, apply the changes by running the following command. Review the Terraform output carefully before typing "yes" to apply the changes. This will issue SSL/TLS certificates. @@ -51,7 +37,7 @@ Run the following command to check the status of a certificate (replace ` --query Certificate.Status ``` -### 4. Update `enable_https = true` in `app-config` +### 3. Update `enable_https = true` in `app-config` **For each application and environment** that should use HTTPS, perform the following. @@ -61,7 +47,7 @@ In each environment config file, set `enable_https` to `true`. This will attach You should have already set `domain_name` as part of [setting up custom domain names](/docs/infra/set-up-network-custom-domains.md). -### 5. Attach certificate to load balancer +### 4. Attach certificate to load balancer **For each application and environment** that should use HTTPS, apply the changes from the previous step by running the following command. Review the Terraform output carefully before typing "yes" to apply the changes. diff --git a/docs/infra/set-up-network.md b/docs/infra/set-up-network.md index f935c3e63..482f5f825 100644 --- a/docs/infra/set-up-network.md +++ b/docs/infra/set-up-network.md @@ -10,27 +10,14 @@ The network set up process will configure and deploy network resources needed fo ## Prerequisites +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. * You have [set up the AWS account(s)](./set-up-aws-accounts.md). * You have [configured all application(s)](./set-up-app-config.md). * You have [configured the project's networks](./set-up-networks.md). ## Instructions -### 1. Make sure you're authenticated into the AWS account you want to configure - -This setup applies to the account you're authenticated into. To see which account that is, run: - -```bash -aws sts get-caller-identity -``` - -To see a more human readable account alias instead of the account, run: - -```bash -aws iam list-account-aliases -``` - -### 2. Configure backend +### 1. Configure backend To create the `.tfbackend` file for the new network, run: @@ -38,7 +25,7 @@ To create the `.tfbackend` file for the new network, run: make infra-configure-network NETWORK_NAME= ``` -### 3. Create network resources +### 2. Create network resources To create the resources, run the following command. Review the Terraform output carefully before typing "yes" to apply the changes. From 71f2cd419a36e296ca9799d3c4b329625affc486 Mon Sep 17 00:00:00 2001 From: Rocket Date: Thu, 16 May 2024 16:14:09 -0700 Subject: [PATCH 89/89] Update command execution docs for consistency --- docs/infra/service-command-execution.md | 41 +++++++++---------------- 1 file changed, 14 insertions(+), 27 deletions(-) diff --git a/docs/infra/service-command-execution.md b/docs/infra/service-command-execution.md index df2ebdfb9..1769991c5 100644 --- a/docs/infra/service-command-execution.md +++ b/docs/infra/service-command-execution.md @@ -6,57 +6,44 @@ The infrastructure supports developer access to a running application's service ## Prerequisites -* You'll need to have [set up infrastructure tools](./set-up-infrastructure-tools.md), like Terraform, AWS CLI, and AWS authentication -* You'll need to have set up the [app environments](./set-up-app-env.md) -* You'll need to have [installed the Session Manager plugin for the AWS CLI](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html) +* You have [set up infrastructure tools](./set-up-infrastructure-tools.md), like Terraform, AWS CLI, and AWS authentication. +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. +* You have [set up the application service](./set-up-app-service.md). +* You have [installed the Session Manager plugin for the AWS CLI](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html). ## Instructions -### 1. Make sure you're authenticated into the AWS account that the ECS container is running in - -This takes effect in whatever account you're authenticated into. To see which account that is, run - -```bash -aws sts get-caller-identity -``` - -To see a more human readable account alias instead of the account, run - -```bash -aws iam list-account-aliases -``` - -### 2. Enable service execution access +### 1. Enable service execution access Within the `app-config` directory (e.g. `infra//app-config`), each environment has its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. In the environment config file for the environment that you want to enable service access, set `enable_command_execution` to `true`. -### 3. Update the network +### 2. Update the network -To enable service execution access, the VPC requires an additional VPC endpoint. Update the network by running +The VPC requires an additional VPC endpoint. To update the network, run: ```bash make infra-update-network NETWORK_NAME= ``` -`ENVIRONMENT` needs to be the name of the network that the application environment is running in. +`` must be the name of the network that the application is running in. -### 4. Update the application service +### 3. Update the application service -To enable service execution access, some configuration changes need to be applied to the ECS Task Definition. Update the service by running +To update the ECS Task Definition to allow command execution, run: ```bash make infra-update-app-service APP_NAME= ENVIRONMENT= ``` -`APP_NAME` needs to be the name of the application folder within the `infra` folder. +`` must be the name of the application folder within the `/infra` folder. -`ENVIRONMENT` needs to be the name of the environment to update. +`` must be the name of the environment to update. -### 5. Execute commands +### 4. Execute commands -To create an interactive shell, run +To create an interactive shell, run: ```bash aws ecs execute-command --cluster \