diff --git a/.github/workflows/template-only-cd.yml b/.github/workflows/template-only-cd.yml index bf9a7c10d..5d7935ba7 100644 --- a/.github/workflows/template-only-cd.yml +++ b/.github/workflows/template-only-cd.yml @@ -34,7 +34,7 @@ jobs: - name: Update infra template working-directory: project-repo - run: ../template-infra/template-only-bin/update-template.sh + run: ../template-infra/template-only-bin/update-template.sh app - name: Push changes to project repo working-directory: project-repo diff --git a/README.md b/README.md index 19715d858..dae7ba17b 100644 --- a/README.md +++ b/README.md @@ -2,63 +2,51 @@ ## Overview -This is a template repository to set up foundational infrastructure for your application in AWS. It is part of a collection of interoperable [Platform templates](https://github.com/navapbc/platform). +This template sets up foundational infrastructure for applications hosted on Amazon Web Services (AWS). It belongs to a collection of interoperable [Platform templates](https://github.com/navapbc/platform). -This template includes setup for: +This template includes: -* **Team workflows** - templates for pull requests (PRs), architecture decision records (ADRs), and Makefiles. -* **Account level foundational infrastructure** - infrastructure for terraform backends, including an S3 bucket and DynamoDB table for storing and managing terraform state files. -* **Application infrastructure** - the infrastructure you need to set up a basic web app, such as a image container repository, load balancer, web service, and database. -* **CI for infra** - GitHub action that performs infra code checks, including linting, validation, and security compliance checks. -* **CD / Deployments** - infrastructure for continuous deployment, including: AWS account access for Github actions, scripts for building and publishing release artifacts, and a Github action for automated deployments from the main branch. -* **Documentation** - technical documentation for the decisions that went into all the defaults that come with the template. +* **Team workflows** - templates for pull requests (PRs), architecture decision records (ADRs), and Makefiles +* **Account level foundational infrastructure** - infrastructure for Terraform backends, including an S3 bucket and DynamoDB table for storing and managing Terraform state files +* **Application infrastructure** - the infrastructure you need to set up a basic web app, including container image repository, load balancer, web service, and database +* **Continuous integration (CI) for infrastructure** - GitHub action that performs infrastructre code checks, including linting, validation, and security compliance checks +* **Continous deployment (CD)** - infrastructure for continuous deployment, including AWS account access for Github actions, scripts for building and publishing release artifacts, and a Github action for automated deployments from the main branch +* **Documentation** - technical documentation for the decisions that went into all the defaults that come with the template -The system architecture will look like this (see [system architecture documentation](/docs/system-architecture.md) for more information): +By default, the system architecture looks like this (for more information, see [system architecture documentation](/docs/system-architecture.md)): ![System architecture](https://lucid.app/publicSegments/view/e5a36152-200d-4d95-888e-4cdbdab80d1b/image.png) ## Application Requirements -This template assumes that you have an application to deploy. See [application requirements](./template-only-docs/application-requirements.md) for more information on what is needed to use the infrastructure template. If you're using one of the [Platform application templates](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), these requirements are already met. +Applications must meet [these requirements](/template-only-docs/application-requirements.md) to be used with this template. If you're using a [Platform application template](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), these requirements are already met. + +### Multiple Applications + +You can use this template with multiple applications. By default, this template assumes your project has one application named `app`. However, it's straightforward to [add additional applications](/template-only-docs/multiple-applications.md). ## Installation -To get started using the infrastructure template on your project, run the following command in your project's directory to execute the [download and install script](https://github.com/navapbc/template-infra/tree/main/template-only-bin/download-and-install-template.sh), which clones the template repository, copies the template files to your repository, and removes any files that are only relevant to the template itself: +This template assumes that you already have an application to deploy. + +To install this template to your project, run the following command in your project's root directory: ```bash curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/download-and-install-template.sh | bash -s ``` +The [download and install script](/template-only-bin/download-and-install-template.sh) clones this template repository, copies the template files to your repository, and removes files that are only relevant to the template itself. + Now you're ready to set up the various pieces of your infrastructure. ## Setup -After downloading and installing the template into your project: +After downloading and installing this template into your project, take the following steps to complete setup: -1. Follow the steps in [infra/README.md](/infra/README.md) to setup the infrastructure for your application. -1. After setting up AWS resources, you can [set up GitHub Actions workflows](./template-only-docs/set-up-ci.md). -1. After configuring GitHub Actions, you can [set up continuous deployment](./template-only-docs/set-up-cd.md). -1. At any point, [set up your team workflow](./template-only-docs/set-up-team-workflow.md). +1. Follow the "First time initialization" steps in [infra/README.md](/infra/README.md). +2. [Set up continuous integration](./template-only-docs/set-up-ci.md). +3. [Set up continuous deployment](./template-only-docs/set-up-cd.md). +4. At any point, [set up your team workflow](./template-only-docs/set-up-team-workflow.md). ## Updates -There are multiple ways to receive template updates on your project. For most updates, you can simply run the [update-template.sh](/template-only-bin/update-template.sh) script - -```bash -curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -``` - -If the update fails the simplest option may be to re-run the installation script above and manually review the changes. - -**Remember:** Make sure to read the release notes in case there are breaking changes you need to address. - -### Renamed applications - -If you renamed your application from `infra/app` to something else like `infra/foo`, then first rename your app back to `infra/app` before applying the updates e.g. - -```bash -mv foo app -mv infra/foo infra/app -curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -mv infra/app infra/foo -mv app foo -``` +This template includes a script to update your project to a newer version of the template. To update your project to a newer version of this template, follow the [update template instructions](/template-only-docs/update-template.md). \ No newline at end of file diff --git a/docs/infra/database-access-control.md b/docs/infra/database-access-control.md index 5b645f93d..c7cbb4d1e 100644 --- a/docs/infra/database-access-control.md +++ b/docs/infra/database-access-control.md @@ -5,7 +5,7 @@ The master user password is managed by Amazon RDS and Secrets Manager. Managing RDS master user passwords with Secrets Manager provides the following security benefits: * RDS rotates database credentials regularly, without requiring application changes. -* Secrets Manager secures database credentials from human access and plain text view. The master password is not even in the terraform state file. +* Secrets Manager secures database credentials from human access and plain text view. The master password is not even in the Terraform state file. For more information about the benefits, see [Benefits of managing master user passwords with Secrets Manager](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-secrets-manager.html#rds-secrets-manager-benefits). diff --git a/docs/infra/destroy-infrastructure.md b/docs/infra/destroy-infrastructure.md index 0635e3302..874f229b4 100644 --- a/docs/infra/destroy-infrastructure.md +++ b/docs/infra/destroy-infrastructure.md @@ -11,7 +11,7 @@ To destroy everything you'll need to undeploy all the infrastructure in reverse $ terraform destroy -var-file=dev.tfvars ``` -2. Then to destroy the backends, first you'll need to add `force_destroy = true` to the S3 buckets, and update the lifecycle block to set `prevent_destroy = false`. Then run `terraform apply` from within the `infra/accounts` directory. The reason we need to do this is because S3 buckets by default are protected from destruction to avoid loss of data. See [Terraform: Destroy/Replace Buckets](https://medium.com/interleap/terraform-destroy-replace-buckets-cf9d63d0029d) for a more in-depth explanation. +2. Then to destroy the backends, first you'll need to add `force_destroy = true` to the S3 buckets, and update the lifecycle block to set `prevent_destroy = false`. Then run `terraform apply` from within the `/infra/accounts` directory. The reason we need to do this is because S3 buckets by default are protected from destruction to avoid loss of data. See [Terraform: Destroy/Replace Buckets](https://medium.com/interleap/terraform-destroy-replace-buckets-cf9d63d0029d) for a more in-depth explanation. ```terraform # infra/modules/modules/terraform-backend-s3/main.tf @@ -46,13 +46,13 @@ To destroy everything you'll need to undeploy all the infrastructure in reverse }2 ``` -4. Then run the following from within the `infra/accounts` directory to copy the `tfstate` back to a local `tfstate` file: +4. Then run the following from within the `/infra/accounts` directory to copy the `tfstate` back to a local `tfstate` file: ```bash terraform init -force-copy ``` -5. Finally, you can run `terraform destroy` within the `infra/accounts` directory. +5. Finally, you can run `terraform destroy` within the `/infra/accounts` directory. ```bash terraform destroy diff --git a/docs/infra/https-support.md b/docs/infra/https-support.md deleted file mode 100644 index aee72d411..000000000 --- a/docs/infra/https-support.md +++ /dev/null @@ -1,40 +0,0 @@ -# HTTPS support - -Production systems will want to use HTTPS rather than HTTP to prevent man-in-the-middle attacks. This document describes how HTTPS is configured. This process will: - -1. Issue an SSL/TLS certificate using Amazon Certificate Manager (ACM) for each domain that we want to support HTTPS -2. Associate the certificate with the application's load balancer so that the load balancer can serve HTTPS requests intended for that domain - -## Requirements - -In order to set up HTTPS support you'll also need to have [set up custom domains](/docs/infra/set-up-custom-domains.md). This is because SSL/TLS certificates must be properly configured for the specific domain to support establishing secure connections. - -## 1. Set desired certificates in domain configuration - -For each custom domain you want to set up in the network, define a certificate configuration object and set the `source` to `issued`. You'll probably want at least one custom domain for each application/service in the network. The custom domain must be either the same as the hosted zone or a subdomain of the hosted zone. - -## 2. Update the network layer to issue the certificates - -Run the following command to issue SSL/TLS certificates for each custom domain you configured - -```bash -make infra-update-network NETWORK_NAME= -``` - -Run the following command to check the status of a certificate (replace `` using the output from the previous command): - -```bash -aws acm describe-certificate --certificate-arn --query Certificate.Status -``` - -## 4. Update `enable_https = true` in `app-config` - -Update `enable_https = true` in your application's `app-config` module. You should have already set `domain_name` as part of [setting up custom domain names](/docs/infra/set-up-custom-domains.md). - -## 5. Attach certificate to load balancer - -Run the following command to attach the SSL/TLS certificate to the load balancer - -```bash -make infra-update-app-service APP_NAME= ENVIRONMENT= -``` diff --git a/docs/infra/intro-to-terraform.md b/docs/infra/intro-to-terraform.md index f91eddfee..bdee976f2 100644 --- a/docs/infra/intro-to-terraform.md +++ b/docs/infra/intro-to-terraform.md @@ -16,7 +16,7 @@ The `terraform destroy` command is a convenient way to destroy all remote object ⚠️ WARNING! ⚠️ This is a destructive command! As a best practice, it's recommended that you comment out resources in non-development environments rather than running this command. `terraform destroy` should only be used as a way to clean up a development environment. e.g. a developer's workspace after they are done with it. -For more information about terraform commands follow the link below: +For more information about Terraform commands follow the link below: - [Basic CLI Features](https://www.terraform.io/cli/commands) diff --git a/docs/infra/making-infra-changes.md b/docs/infra/making-infra-changes.md index 5989134e6..e8f125042 100644 --- a/docs/infra/making-infra-changes.md +++ b/docs/infra/making-infra-changes.md @@ -2,7 +2,7 @@ ## Requirements -First read [Module Architecture](./module-architecture.md) to understand how the terraform modules are structured. +First read [Module Architecture](./module-architecture.md) to understand how the Terraform modules are structured. ## Using make targets (recommended) @@ -34,9 +34,9 @@ TF_CLI_ARGS_apply='-input=false -auto-approve' make infra-update-app-service APP TF_CLI_ARGS_apply='-var=image_tag=abcdef1' make infra-update-app-service APP_NAME=app ENVIRONMENT=dev ``` -## Using terraform CLI wrapper scripts +## Using Terraform CLI wrapper scripts -An alternative to using the Makefile is to directly use the terraform wrapper scripts that the Makefile uses: +An alternative to using the Makefile is to directly use the Terraform wrapper scripts that the Makefile uses: ```bash project-root$ ./bin/terraform-init.sh app/service dev @@ -48,7 +48,7 @@ Look in the script files for more details on usage. ## Using Terraform CLI directly -Finally, if the wrapper scripts don't meet your needs, you can always run `terraform` directly from the root module directory. You may need to do this if you are running terraform commands other than `terraform plan` and `terraform apply`, such as `terraform import`, `terraform taint`, etc. To do this, you'll need to pass in the appropriate `tfvars` and `tfbackend` files to `terraform init` and `terraform apply`. For example, to make changes to the application's service resources in the dev environment, cd to the `infra/app/service` directory and run: +Finally, if the wrapper scripts don't meet your needs, you can always run `terraform` directly from the root module directory. You may need to do this if you are running Terraform commands other than `terraform plan` and `terraform apply`, such as `terraform import`, `terraform taint`, etc. To do this, you'll need to pass in the appropriate `.tfbackend` files to `terraform init` and `terraform apply`. For example, to make changes to the application's service resources in the dev environment, change to the `/infra/app/service` directory and run: ```bash infra/app/service$ terraform init -backend-config=dev.s3.tfbackend diff --git a/docs/infra/module-architecture.md b/docs/infra/module-architecture.md index 6777faa90..5ad0a2c28 100644 --- a/docs/infra/module-architecture.md +++ b/docs/infra/module-architecture.md @@ -85,7 +85,7 @@ When deciding which layer to put an infrastructure resource in, follow the follo * **Consider policy constraints on what resources the project team is authorized to manage:** Different categories of resources may have different requirements on who is allowed to create and manage those resources. Resources that the project team are not allowed to manage directly should not be mixed with resources that the project team needs to manage directly. -* **Consider out-of-band dependencies:** Put infrastructure resources that require steps outside of terraform to be completed configured in layers that are upstream to resources that depend on those completed resources. For example, after creating a database cluster, the database schemas, roles, and privileges need to be configured before they can be used by a downstream service. Therefore database resources should be separate from the service layer so that the database can be configured fully before attempting to create the service layer resources. +* **Consider out-of-band dependencies:** Put infrastructure resources that require steps outside of Terraform to be completed configured in layers that are upstream to resources that depend on those completed resources. For example, after creating a database cluster, the database schemas, roles, and privileges need to be configured before they can be used by a downstream service. Therefore database resources should be separate from the service layer so that the database can be configured fully before attempting to create the service layer resources. ## Making changes to infrastructure diff --git a/docs/infra/service-command-execution.md b/docs/infra/service-command-execution.md index df2ebdfb9..1769991c5 100644 --- a/docs/infra/service-command-execution.md +++ b/docs/infra/service-command-execution.md @@ -6,57 +6,44 @@ The infrastructure supports developer access to a running application's service ## Prerequisites -* You'll need to have [set up infrastructure tools](./set-up-infrastructure-tools.md), like Terraform, AWS CLI, and AWS authentication -* You'll need to have set up the [app environments](./set-up-app-env.md) -* You'll need to have [installed the Session Manager plugin for the AWS CLI](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html) +* You have [set up infrastructure tools](./set-up-infrastructure-tools.md), like Terraform, AWS CLI, and AWS authentication. +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. +* You have [set up the application service](./set-up-app-service.md). +* You have [installed the Session Manager plugin for the AWS CLI](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html). ## Instructions -### 1. Make sure you're authenticated into the AWS account that the ECS container is running in - -This takes effect in whatever account you're authenticated into. To see which account that is, run - -```bash -aws sts get-caller-identity -``` - -To see a more human readable account alias instead of the account, run - -```bash -aws iam list-account-aliases -``` - -### 2. Enable service execution access +### 1. Enable service execution access Within the `app-config` directory (e.g. `infra//app-config`), each environment has its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. In the environment config file for the environment that you want to enable service access, set `enable_command_execution` to `true`. -### 3. Update the network +### 2. Update the network -To enable service execution access, the VPC requires an additional VPC endpoint. Update the network by running +The VPC requires an additional VPC endpoint. To update the network, run: ```bash make infra-update-network NETWORK_NAME= ``` -`ENVIRONMENT` needs to be the name of the network that the application environment is running in. +`` must be the name of the network that the application is running in. -### 4. Update the application service +### 3. Update the application service -To enable service execution access, some configuration changes need to be applied to the ECS Task Definition. Update the service by running +To update the ECS Task Definition to allow command execution, run: ```bash make infra-update-app-service APP_NAME= ENVIRONMENT= ``` -`APP_NAME` needs to be the name of the application folder within the `infra` folder. +`` must be the name of the application folder within the `/infra` folder. -`ENVIRONMENT` needs to be the name of the environment to update. +`` must be the name of the environment to update. -### 5. Execute commands +### 4. Execute commands -To create an interactive shell, run +To create an interactive shell, run: ```bash aws ecs execute-command --cluster \ diff --git a/docs/infra/set-up-app-build-repository.md b/docs/infra/set-up-app-build-repository.md index 01c32a634..9652fe773 100644 --- a/docs/infra/set-up-app-build-repository.md +++ b/docs/infra/set-up-app-build-repository.md @@ -1,32 +1,36 @@ # Set up application build repository +Follow these instructions for **each application** in your project (you can have one or more in your project). If the application does not need a build repository, skip to the bottom of this document. + The application build repository setup process will create infrastructure resources needed to store built release-candidate artifacts used to deploy the application to an environment. -## Requirements +## Prerequisites -Before setting up the application's build repository you'll need to have: +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You have [configured the application](/infra/app/app-config/main.tf). +* You have [set up the network(s)](./set-up-networks.md). -1. [Set up the AWS account](./set-up-aws-account.md) -2. [Configure the app](/infra/app/app-config/main.tf) +## Instructions -## 1. Configure backend +### 1. Configure backend -To create the `tfbackend` file for the build repository using the backend configuration values from your current AWS account, run +To create the `.tfbackend` file for the build repository, run: ```bash -make infra-configure-app-build-repository APP_NAME=app +make infra-configure-app-build-repository APP_NAME= ``` -Pass in the name of the app folder within `infra`. By default this is `app`. +`` must be the name of the application folder within the `/infra` folder. -## 2. Create build repository resources +### 2. Create build repository resources -Now run the following commands to create the resources, making sure to verify the plan before confirming the apply. +To create the resources, run the following command. Review the Terraform output carefully before typing "yes" to apply the changes. ```bash -make infra-update-app-build-repository APP_NAME=app +make infra-update-app-build-repository APP_NAME= ``` -## Set up application environments +## If the application does not need a build repository -Once you set up the deployment process, you can proceed to [set up application environments](./set-up-app-env.md) +If the application does not need a build repository, such as if the project uses pre-built images hosted in an external container repository, delete the application's build repository module (e.g. `/infra//build-repository`). \ No newline at end of file diff --git a/docs/infra/set-up-app-config.md b/docs/infra/set-up-app-config.md new file mode 100644 index 000000000..c93413d42 --- /dev/null +++ b/docs/infra/set-up-app-config.md @@ -0,0 +1,45 @@ +# Set up application config + +Follow these instructions for **each application** in your project (you can have one or more in your project). + +## Prerequisites + +* You have [configured the project](/infra/project-config/main.tf). +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). + +## Instructions + +### 1. (Optional) Rename the application + +By default, the application module is named `app` in [`/infra`](/infra/). You may want to rename the application to something project-specific. + +### 2. Configure app-config + +Modify the following values in the application's `app-config/main.tf` (e.g. `/infra//app-config/main.tf`): + +* Set the `environments` array to list the names of the environments for this application. By default, this is set to `["dev", "staging", "prod"]`. +* Set `has_database` to `true` or `false` to indicate whether or not the application relies on a database. This setting determines whether or not to create a database and create VPC endpoints needed by the database layer. By default, this is set to `false`. +* Set `has_external_non_aws_service` to `true` or `false` to indicate whether or not your application makes calls to an external non-AWS service. This setting determines whether or not to create NAT gateways, which allows the service in the private subnet to make requests to the internet. By default, this is set to `false`. +* Set `has_incident_management_service` to `true` or `false` to indicate whether the application should integrate with an incident management service. By default, this is set to `false`. +* Set the `account_names_by_environment` hash to map environments to AWS accounts. Use the mapping you decided on in the [set up AWS accounts](./set-up-aws-accounts.md) step. + +To use [feature flags](/docs/feature-flags.md), modify the values in the application's `app-config/feature-flags.tf` (e.g. `/infra//app-config/feature-flags.tf`). + +### 3. Configure each environment + +Within the application's `app-config` directory (e.g. `/infra//app-config`), each environment configured in the `environments` array in the previous step needs its own config file. For example, if the application has three environments `dev`, `staging`, and `prod`, there must be corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. + +In each environment config file, modify the following values: + +* Set `environment` to the name of the environment. This should match the name of the file. +* Set `network_name`. By default, it should match the name of the environment. This mapping ensures that each network is configured appropriately based on the application(s) in that network (see `local.apps_in_network` in `/infra/networks/main.tf`). Failure to set the network name properly may cause the network layer to use incorrect application configurations for `has_database` and `has_external_non_aws_service`. +* Skip `domain_name` for now. +* Skip `enable_https` for now. + +When configuring the production environment, update these settings based on your project's needs: + +* `service_cpu` +* `service_memory` +* `service_desired_instance_count` + +Consider doing a load test if your application is sensitive to performance. diff --git a/docs/infra/set-up-app-database.md b/docs/infra/set-up-app-database.md new file mode 100644 index 000000000..fb9567b6d --- /dev/null +++ b/docs/infra/set-up-app-database.md @@ -0,0 +1,96 @@ +# Set up database + +Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. If the application does not need a database, skip to the bottom of this document. + +The database set up process will: + +* Configure and deploy an application database cluster using [Amazon Aurora Serverless V2](https://aws.amazon.com/rds/aurora/serverless/) +* Create a [PostgreSQL schema](https://www.postgresql.org/docs/current/ddl-schemas.html) called `app` to contain tables used by the application +* Create an IAM policy that allows IAM roles with that policy attached to [connect to the database using IAM authentication](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.html) +* Create an [AWS Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) called "role manager" to provision the [PostgreSQL database users](https://www.postgresql.org/docs/8.0/user-manag.html) used by the application service and the migrations task +* Invoke the role manager function to create Postgres users called `app` and `migrator` + +## Prerequisites + +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You have [configured the application](/infra/app/app-config/main.tf). +* You have [set up the network(s)](./set-up-networks.md). +* You have [pip](https://pypi.org/project/pip/) installed (pip is needed to download dependencies for the role manager Lambda function). + +## Instructions + +### 1. Configure backend + +To create the `.tfbackend` file for the new application environment, run: + +```bash +make infra-configure-app-database APP_NAME= ENVIRONMENT= +``` + +`` must be the name of the application folder within the `/infra` folder. + +`` must be the name of the environment to update. This will create a file called `.s3.tfbackend` in `/infra//database`. + +### 2. Create database resources + +To create the resources, run the following command. Review the Terraform output carefully before typing "yes" to apply the changes. This can take over 5 minutes. + +```bash +make infra-update-app-database APP_NAME= ENVIRONMENT= +``` + +### 3. Create Postgres users + +To trigger the role manager Lambda function that was created in the previous step to create the `app` and `migrator` Postgres users, run: + +```bash +make infra-update-app-database-roles APP_NAME= ENVIRONMENT= +``` + +The Lambda function's response should describe the resulting PostgreSQL roles and groups that are configured in the database. It should look like a minified version of the following: + +```json +{ + "roles": [ + "postgres", + "migrator", + "app" + ], + "roles_with_groups": { + "rds_superuser": "rds_password", + "pg_monitor": "pg_read_all_settings,pg_read_all_stats,pg_stat_scan_tables", + "postgres": "rds_superuser", + "app": "rds_iam", + "migrator": "rds_iam" + }, + "schema_privileges": { + "public": "{postgres=UC/postgres,=UC/postgres}", + "app": "{migrator=UC/migrator,app=U/migrator}" + } +} +``` + +#### Important note on Postgres table permissions + +Before creating migrations that create tables, first create a migration that includes the following SQL command (or equivalent if your migrations are written in a general-purpose programming language): + +```sql +ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO app +``` + +This will cause all future tables created by the `migrator` user to automatically be accessible by the `app` user. See the [Postgres docs on ALTER DEFAULT PRIVILEGES](https://www.postgresql.org/docs/current/sql-alterdefaultprivileges.html) for more info. As an example see the example app's migrations file [migrations.sql](https://github.com/navapbc/template-infra/blob/main/app/migrations.sql). + +Why is this needed? The reason is that the `migrator` role will be used by the migration task to run database migrations (creating tables, altering tables, etc.), while the `app` role will be used by the web service to access the database. Moreover, in Postgres, new tables won't automatically be accessible by roles other than the creator unless specifically granted, even if those other roles have usage access to the schema that the tables are created in. In other words, if the `migrator` user created a new table `foo` in the `app` schema, the `app` user will not automatically be able to access it by default. + +### 4. Check that database roles have been configured properly + +Verify the that the database roles have been configured properly by running: + +```bash +make infra-check-app-database-roles APP_NAME= ENVIRONMENT= +``` + +## If the application does not need a database + +If the application does not need a database, such as if the project uses an alternative for data persistence, delete the application's database module (e.g. `/infra//database`) and set the application's `app-config` setting `has_database` to `false` (see [set up app config](./set-up-app-config.md)). \ No newline at end of file diff --git a/docs/infra/set-up-app-env.md b/docs/infra/set-up-app-env.md deleted file mode 100644 index 31a588f8c..000000000 --- a/docs/infra/set-up-app-env.md +++ /dev/null @@ -1,60 +0,0 @@ -# Set up Application Environment - -The application environment setup process will: - -1. Configure a new application environment and create the infrastructure resources for the application in that environment - -## Requirements - -Before setting up the application's environments you'll need to have: - -1. [A compatible application in the app folder](https://github.com/navapbc/template-infra/blob/main/template-only-docs/application-requirements.md) -2. [Configure the app](/infra/app/app-config/main.tf). Make sure you update `has_database` to `true` or `false` depending on whether or not your application has a database to integrate with. - 1. If you're configuring your production environment, make sure to update the `service_cpu`, `service_memory`, and `service_desired_instance_count` settings based on the project's needs. If your application is sensitive to performance, consider doing a load test. -3. [Create a nondefault VPC to be used by the application](./set-up-network.md) -4. (If the application has a database) [Set up the database for the application](./set-up-database.md) -5. (If you have an incident management service) [Set up monitoring](./set-up-monitoring-alerts.md) -6. [Set up the application build repository](./set-up-app-build-repository.md) - -## 1. Configure backend - -To create the `tfbackend` and `tfvars` files for the new application environment, run - -```bash -make infra-configure-app-service APP_NAME=app ENVIRONMENT= -``` - -`APP_NAME` needs to be the name of the application folder within the `infra` folder. It defaults to `app`. -`ENVIRONMENT` needs to be the name of the environment you are creating. This will create a file called `.s3.tfbackend` in the `infra/app/service` module directory. - -Depending on the value of `has_database` in the [app-config module](/infra/app/app-config/main.tf), the application will be configured with or without database access. - -## 2. Build and publish the application to the application build repository - -Before creating the application resources, you'll need to first build and publish at least one image to the application build repository. - -There are two ways to do this: - -1. Trigger the "Build and Publish" workflow from your repo's GitHub Actions tab. This option requires that the `role-to-assume` GitHub workflow variable has already been set up as part of the overall infra account setup process. -1. Alternatively, run the following from the root directory. This option can take much longer than the GitHub workflow, depending on your machine's architecture. - - ```bash - make release-build APP_NAME=app - make release-publish APP_NAME=app - ``` - -Copy the image tag name that was published. You'll need this in the next step. - -## 3. Create application resources with the image tag that was published - -Now run the following commands to create the resources, using the image tag that was published in the previous step. Review the terraform before confirming "yes" to apply the changes. - -```bash -TF_CLI_ARGS_apply="-var=image_tag=" make infra-update-app-service APP_NAME=app ENVIRONMENT= -``` - -## 4. Configure monitoring alerts - -Configure email alerts, external incident management service integration and additional Cloudwatch Alerts. -[Configure monitoring module](./set-up-monitoring-alerts.md) - diff --git a/docs/infra/set-up-app-monitoring-alerts.md b/docs/infra/set-up-app-monitoring-alerts.md new file mode 100644 index 000000000..c021acf9b --- /dev/null +++ b/docs/infra/set-up-app-monitoring-alerts.md @@ -0,0 +1,71 @@ +# Set up monitoring notifications + +Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. If the application does not need a monitoring notifications, skip to the bottom of this document. + +The monitoring module defines metric-based alerting policies that provides awareness into issues with the cloud application. The module supports integration with external incident management tools like Splunk-On-Call or Pagerduty. It also supports email alerts. + +## Prerequisites + +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You have [configured the application](/infra/app/app-config/main.tf). +* You have [set up the network(s)](./set-up-networks.md). +* If you need a container build repository, you have [set up the build repository](./set-up-app-build-repository.md). + +## Instructions to set up email alerts + +When any of the alerts described by the module are triggered, a notification will be sent to all emails specified in `email_alerts_subscription_list`. + +### 1. Update the application's service layer + +In the application's service module (e.g. `/infra//service/main.tf`), uncomment the `email_alerts_subscription_list` key and add the emails that should be notified. + +For example: + +``` +module "monitoring" { + source = "../../modules/monitoring" + email_alerts_subscription_list = ["email1@email.com", "email2@email.com"] + ... +} +``` + +### 2. Update the application service + +To apply the changes, run the following command. Review the Terraform output carefully before typing "yes" to apply the changes. + +```bash +make infra-update-app-service APP_NAME= ENVIRONMENT= +``` + +`` must be the name of the application folder within the `/infra` folder. + +`` must be the name of the environment to update. + +## Instructions to set up external incident management service integration + +### 1. Update the application config + +In the application's `app-config/main.tf` (e.g. `/infra//app-config/main.tf`), set `has_incident_management_service` to `true`. + +### 2. Add the external url as a secret + +Get the integration URL for the incident management service and run the following command to store it in AWS SSM Parameter Store: + +```bash +make infra-configure-monitoring-secrets APP_NAME= ENVIRONMENT= URL= +``` + +### 3. Update the application service + +To apply the changes, run the following command. Review the Terraform output carefully before typing "yes" to apply the changes. + +```bash +make infra-update-app-service APP_NAME= ENVIRONMENT= +``` + +## If the application does not need monitoring notifications + +If the application does not need monitoring notifications, complete the following steps: + +1. In the application's service module (e.g. `/infra//service/main.tf`), comment out `email_alerts_subscription_list`. +2. In the application's `app-config` (e.g. `/infra//app-config/main.tf`), set `has_incident_management_service` to `false`. \ No newline at end of file diff --git a/docs/infra/set-up-app-service.md b/docs/infra/set-up-app-service.md new file mode 100644 index 000000000..8aaeacb07 --- /dev/null +++ b/docs/infra/set-up-app-service.md @@ -0,0 +1,55 @@ +# Set up application service + +Follow these instructions for **each application** (you can have one or more in your project) and **each environment** in your project. + +The application service set up process will: + +* Configure an ECS Fargate Service and Task to host the application +* Create a load balancer for the application +* Create an S3 bucket for general object storage +* Set up CloudWatch for logging, monitoring, and alerts +* Configure CloudWatch Evidently to support feature flags + +## Prerequisites + +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You have [configured the application](/infra/app/app-config/main.tf). +* You have [set up the network(s)](./set-up-networks.md). +* If you need a container build repository, you have [set up the build repository](./set-up-app-build-repository.md). +* If you need a database for the application, you have [set up the database](./set-up-app-database.md). + +## Instructions + +### 1. Configure backend + +To create the `.tfbackend` files for the new application service, run: + +```bash +make infra-configure-app-service APP_NAME= ENVIRONMENT= +``` + +`` must be the name of the application folder within the `/infra` folder. + +`` must be the name of the environment to update. This will create a file called `.s3.tfbackend` in `/infra//service`. + +### 2. Build and publish the application to the build repository + +Before creating the application resources, you need to build and publish at least one image to the build repository. This step does not need to be run per environment. + +Run the following commands from the project's root directory: + +```bash +make release-build APP_NAME= +make release-publish APP_NAME= +``` + +Copy the image tag name that is output. You'll need this in the next step. + +### 3. Create application resources with the image tag that was published + +To create the resources, run the following command using the image tag output by the previous step. Review the Terraform output carefully before typing "yes" to apply the changes. This can take over 5 minutes. + +```bash +TF_CLI_ARGS_apply="-var=image_tag=" make infra-update-app-service APP_NAME= ENVIRONMENT= +``` \ No newline at end of file diff --git a/docs/infra/set-up-aws-account.md b/docs/infra/set-up-aws-account.md index 47af9dbcc..2850fb0aa 100644 --- a/docs/infra/set-up-aws-account.md +++ b/docs/infra/set-up-aws-account.md @@ -1,49 +1,35 @@ # Set up AWS account +Follow these instructions for **each AWS account** you want to configure (you can have one or more in your project). + The AWS account setup process will: -1. Create the [Terraform backend](https://www.terraform.io/language/settings/backends/configuration) resources needed to store Terraform's infrastructure state files. The project uses an [S3 backend](https://www.terraform.io/language/settings/backends/s3). -2. Create the [OpenID connect provider in AWS](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html) to allow GitHub Actions to access AWS account resources. -3. Create the IAM role and policy that GitHub Actions will use to manage infrastructure resources. +* Create the [Terraform backend](https://www.terraform.io/language/settings/backends/configuration) resources needed to store Terraform's infrastructure state files using an [S3 backend](https://www.terraform.io/language/settings/backends/s3). +* Create the [OpenID connect provider in AWS](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html) to allow GitHub Actions to access AWS account resources. +* Create the IAM role and policy that GitHub Actions will use to manage infrastructure resources. ## Prerequisites -* You'll need to have [set up infrastructure tools](./set-up-infrastructure-tools.md), like Terraform, AWS CLI, and AWS authentication. -* You'll also need to make sure the [project is configured](/infra/project-config/main.tf). +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. +* You have [set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md). +* You have [configured the project](/infra/project-config/main.tf). +* You have [decided on your environment and AWS account strategy](./set-up-aws-accounts.md). ## Instructions -### 1. Make sure you're authenticated into the AWS account you want to configure - -The account set up sets up whatever account you're authenticated into. To see which account that is, run - -```bash -aws sts get-caller-identity -``` - -To see a more human readable account alias instead of the account, run +### 1. Create backend resources and `.tfbackend` config file -```bash -aws iam list-account-aliases -``` - -### 2. Create backend resources and tfbackend config file - -Run the following command, replacing `` with a human readable name for the AWS account that you're authenticated into. The account name will be used to prefix the created tfbackend file so that it's easier to visually identify as opposed to identifying the file using the account id. For example, you have an account per environment, the account name can be the name of the environment (e.g. "prod" or "staging"). Or if you are setting up an account for all lower environments, account name can be "lowers". If your AWS account has an account alias, you can also use that. +Run the following command, replacing `` with a human readable name for the AWS account that you're authenticated into. The account name will be used to prefix the `.tfbackend` file. For example, if you have an account per environment, the account name can be the name of the environment (e.g. "prod" or "staging"). Or if you are setting up an account for all lower environments, the account name can be "lowers". If your AWS account has an account alias, you can also use that. ```bash make infra-set-up-account ACCOUNT_NAME= ``` -This command will create the S3 tfstate bucket and the GitHub OIDC provider. It will also create a `[account name].[account id].s3.tfbackend` file in the `infra/accounts` directory. - -### 3. Update the account names map in app-config - -In [app-config/main.tf](/infra/app/app-config/main.tf), update the `account_names_by_environment` config to reflect the account name you chose. +This command will create the S3 tfstate bucket and the GitHub OIDC provider. It will also create a `[account name].[account id].s3.tfbackend` file in the `/infra/accounts` directory. -## Making changes to the account +## Making changes to an AWS account -If you make changes to the account terraform and want to apply those changes, run +If you make changes to an account, apply those changes by running: ```bash make infra-update-current-account diff --git a/docs/infra/set-up-aws-accounts.md b/docs/infra/set-up-aws-accounts.md new file mode 100644 index 000000000..aa0d32459 --- /dev/null +++ b/docs/infra/set-up-aws-accounts.md @@ -0,0 +1,69 @@ +# Set up AWS accounts + +The AWS accounts setup process will: + +* Help you decide on your environments +* Help you decide on your AWS account strategy +* Configure all of your AWS accounts + +## Prerequisites + +* You have [set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md). + +## Instructions + +### 1. Decide on environments + +The names of your environments must be consistent for all applications. By default, the environments are: + +* `dev` +* `staging` +* `prod` + +You can changes these names, add additional environments, or delete any these environments. You must have at least one environment. You can always add new environments or delete existing environments later. + +### 2. Decide on AWS account strategy + +This template supports managing resources in multiple [AWS accounts](https://docs.aws.amazon.com/accounts/latest/reference/accounts-welcome.html). To do so, each environment (e.g. `dev`, `staging`, `prod`) is mapped to a specified AWS account. Resources that are shared across environments, such as build repositories, are also explicitly mapped to a specified AWS account. In the following diagrams, these are represented by the box labeled `shared`. + +The environment-to-AWS-account mapping is specified for each application. Multiple applications can share AWS accounts. + +A simple project might have only one AWS account and all environments should be deployed to this environment: + +```mermaid +flowchart TD + shared --> my_aws_account + dev --> my_aws_account + staging --> my_aws_account + prod --> my_aws_account +``` + +A more complex project might have separate AWS accounts for environment, enhancing security by isolating each environment into completely separate AWS accounts: + +```mermaid +flowchart TD + shared --> shared_aws_account + dev --> dev_aws_account + staging --> staging_aws_account + prod --> prod_aws_account +``` + +A project could also isolate just the `prod` environment and group the lower environments: + +```mermaid +flowchart TD + shared --> shared_aws_account + dev --> lower_aws_account + staging --> lower_aws_account + prod --> prod_aws_account +``` + +Decide on the strategy that is appropriate for your project. + +### 3. Ensure AWS account(s) have been created + +For **each AWS account** you wish to use, ensure the AWS account has been created and you are able to authenticate to it. + +### 4. Set up AWS account + +For **each AWS account** you wish to use, [set up the AWS account](./set-up-aws-account.md). \ No newline at end of file diff --git a/docs/infra/set-up-custom-domains.md b/docs/infra/set-up-custom-domains.md deleted file mode 100644 index e4ff84740..000000000 --- a/docs/infra/set-up-custom-domains.md +++ /dev/null @@ -1,73 +0,0 @@ -# Custom domains - -Production systems will want to set up custom domains to route internet traffic to their application services rather than using AWS-generated hostnames for the load balancers or the CDN. This document describes how to configure custom domains. The custom domain setup process will: - -1. Create an [Amazon Route 53 hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) to manage DNS records for a domain and subdomains -2. Create a DNS A (address) records to route traffic from a custom domain to the application's load balancer - -## Requirements - -Before setting up custom domains you'll need to have [set up the AWS account](./set-up-aws-account.md) - -## 1. Set hosted zone in domain configuration - -Update the value for the `hosted_zone` in the domain configuration. The custom domain configuration is defined as a `domain_config` object in the [network section of the project config module](/infra/project-config/networks.tf). A [hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) represents a domain and all of its subdomains. For example, a hosted zone of `platform-test.navateam.com` includes `platform-test.navateam.com`, `cdn.platform-test.navateam.com`, `notifications.platform-test.navateam.com`, `foo.bar.platform-test.navateam.com`, etc. - -## 2. Update the network layer to create the hosted zone - -Run the following command to create the hosted zone specified in the domain configuration. - -```bash -make infra-update-network NETWORK_NAME= -``` - -## 3. Delegate DNS requests to the newly created hosted zone - -You most likely registered your domain outside of this project. Using whichever service you used to register the domain name (e.g. Namecheap, GoDaddy, Google Domains, etc.), add a DNS NS (nameserver) record. Set the "name" equal to the `hosted_zone` and set the value equal to the list of hosted zone name servers that was created in the previous step. You can see the list of servers by running - -```bash -terraform -chdir=infra/networks output -json hosted_zone_name_servers -``` - -Your NS record might look something like this: - -**Name**: - -```text -platform-test.navateam.com -``` - -**Value**: (Note the periods after each of the server addresses) - -```text -ns-1431.awsdns-50.org. -ns-1643.awsdns-13.co.uk. -ns-687.awsdns-21.net. -ns-80.awsdns-10.com. -``` - -Run the following command to verify that DNS requests are being served by the hosted zone nameservers using `nslookup`. - -```bash -nslookup -type=NS -``` - -## 4. Configure custom domain for your application - -Define the `domain_name` for each of the application environments in the `app-config` module. The `domain_name` must be either the same as the `hosted_zone` or a subdomain of the `hosted_zone`. For example, if your hosted zone is `platform-test.navateam.com`, then `platform-test.navateam.com` and `cdn.platform-test.navateam.com` are both valid values for `domain_name`. - -## 5. Create A (address) records to route traffice from the custom domain to your application's load balancer - -Run the following command to create the A record that routes traffic from the custom domain to the application's load balancer. - -```bash -make infra-update-app-service APP_NAME= ENVIRONMENT= -``` - -## 6. Repeat for each application - -If you have multiple applications in the same network, repeat steps 4 and 5 for each application. - -## Externally managed DNS - -If you plan to manage DNS records outside of the project, then set `network_configs[*].domain_config.manage_dns = false` in [the networks section of the project-config module](/infra/project-config/networks.tf). diff --git a/docs/infra/set-up-database.md b/docs/infra/set-up-database.md deleted file mode 100644 index 7f07c4518..000000000 --- a/docs/infra/set-up-database.md +++ /dev/null @@ -1,88 +0,0 @@ -# Set up database - -The database setup process will: - -1. Configure and deploy an application database cluster using [Amazon Aurora Serverless V2](https://aws.amazon.com/rds/aurora/serverless/) -2. Create a [PostgreSQL schema](https://www.postgresql.org/docs/current/ddl-schemas.html) `app` to contain tables used by the application. -3. Create an IAM policy that allows IAM roles with that policy attached to [connect to the database using IAM authentication](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.html) -4. Create an [AWS Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html), the "role manager", for provisioning the [PostgreSQL database users](https://www.postgresql.org/docs/8.0/user-manag.html) that will be used by the application service and by the migrations task. -5. Invoke the role manager function to create the `app` and `migrator` Postgres users. - -## Requirements - -Before setting up the database you'll need to have: - -1. [Set up the AWS account](./set-up-aws-account.md) -2. pip installed (pip is needed to download dependencies for the role manager Lambda function) - -## 1. Configure backend - -To create the `tfbackend` file for the new application environment, run - -```bash -make infra-configure-app-database APP_NAME= ENVIRONMENT= -``` - -`APP_NAME` needs to be the name of the application folder within the `infra` folder. By default, this is `app`. -`ENVIRONMENT` needs to be the name of the environment you are creating. This will create a file called `.s3.tfbackend` in the `infra/app/service` module directory. - -## 2. Create database resources - -Now run the following commands to create the resources. Review the terraform before confirming "yes" to apply the changes. This can take over 5 minutes. - -```bash -make infra-update-app-database APP_NAME=app ENVIRONMENT= -``` - -## 3. Create Postgres users - -Trigger the role manager Lambda function that was created in the previous step to create the application and `migrator` Postgres users. - -```bash -make infra-update-app-database-roles APP_NAME=app ENVIRONMENT= -``` - -The Lambda function's response should describe the resulting PostgreSQL roles and groups that are configured in the database. It should look like a minified version of the following: - -```json -{ - "roles": [ - "postgres", - "migrator", - "app" - ], - "roles_with_groups": { - "rds_superuser": "rds_password", - "pg_monitor": "pg_read_all_settings,pg_read_all_stats,pg_stat_scan_tables", - "postgres": "rds_superuser", - "app": "rds_iam", - "migrator": "rds_iam" - }, - "schema_privileges": { - "public": "{postgres=UC/postgres,=UC/postgres}", - "app": "{migrator=UC/migrator,app=U/migrator}" - } -} -``` - -### Important note on Postgres table permissions - -Before creating migrations that create tables, first create a migration that includes the following SQL command (or equivalent if your migrations are written in a general-purpose programming language): - -```sql -ALTER DEFAULT PRIVILEGES GRANT ALL ON TABLES TO app -``` - -This will cause all future tables created by the `migrator` user to automatically be accessible by the `app` user. See the [Postgres docs on ALTER DEFAULT PRIVILEGES](https://www.postgresql.org/docs/current/sql-alterdefaultprivileges.html) for more info. As an example see the example app's migrations file [migrations.sql](https://github.com/navapbc/template-infra/blob/main/app/migrations.sql). - -Why is this needed? The reason is that the `migrator` role will be used by the migration task to run database migrations (creating tables, altering tables, etc.), while the `app` role will be used by the web service to access the database. Moreover, in Postgres, new tables won't automatically be accessible by roles other than the creator unless specifically granted, even if those other roles have usage access to the schema that the tables are created in. In other words, if the `migrator` user created a new table `foo` in the `app` schema, the `app` user will not automatically be able to access it by default. - -## 4. Check that database roles have been configured properly - -```bash -make infra-check-app-database-roles APP_NAME=app ENVIRONMENT= -``` - -## Set up application environments - -Once you set up the deployment process, you can proceed to [set up the application service](./set-up-app-env.md) diff --git a/docs/infra/environment-variables-and-secrets.md b/docs/infra/set-up-environment-variables-and-secrets.md similarity index 97% rename from docs/infra/environment-variables-and-secrets.md rename to docs/infra/set-up-environment-variables-and-secrets.md index f05d39df6..d6bb61d69 100644 --- a/docs/infra/environment-variables-and-secrets.md +++ b/docs/infra/set-up-environment-variables-and-secrets.md @@ -1,5 +1,7 @@ # Environment variables and secrets +Follow these instructions for **each application** in your project (you can have one or more in your project). + Applications follow [12-factor app](https://12factor.net/) principles to [store configuration as environment variables](https://12factor.net/config). The infrastructure provides some of these environment variables automatically, such as environment variables to authenticate as the ECS task role, environment variables for database access, and environment variables for accessing document storage. However, many applications require extra custom environment variables for application configuration and for access to secrets. This document describes how to configure application-specific environment variables and secrets. It also describes how to override those environment variables for a specific environment. ## Application-specific extra environment variables diff --git a/docs/infra/set-up-infrastructure-tools.md b/docs/infra/set-up-infrastructure-tools.md index 1bf076ad0..7cb9b51bf 100644 --- a/docs/infra/set-up-infrastructure-tools.md +++ b/docs/infra/set-up-infrastructure-tools.md @@ -1,42 +1,51 @@ # Set up infrastructure developer tools -If you are contributing to infrastructure, you will need to complete these setup steps. +Complete these steps to work on the infrastructure. ## Prerequisites -### Install Terraform +* None -[Terraform](https://www.terraform.io/) is an infrastructure as code (IaC) tool that allows you to build, change, and version infrastructure safely and efficiently. This includes both low-level components like compute instances, storage, and networking, as well as high-level components like DNS entries and SaaS features. +## Instructions -You may need different versions of Terraform since different projects may require different versions of Terraform. The best way to manage Terraform versions is with [Terraform Version Manager](https://github.com/tfutils/tfenv). +### Install Terraform -To install via [Homebrew](https://brew.sh/) +[Terraform](https://www.terraform.io/) is an infrastructure as code (IaC) tool that allows you to build, change, and version infrastructure safely and efficiently. This includes both low-level components, like compute instances, storage, and networking, as well as high-level components, like DNS entries and Software-as-a-Service (SaaS) features. -```bash -brew install tfenv -``` +You may need to install different versions of terraform on your machine because different projects may require different versions. We recommend managing terraform with [Terraform Version Manager (tfenv)](https://github.com/tfutils/tfenv). -Then install the version of Terraform you need. - -```bash -tfenv install 1.4.6 -``` +1. Use [Homebrew](https://brew.sh/) to install tfenv: + ```bash + brew install tfenv + ``` +2. Install the version of Terraform you need: + ```bash + tfenv install 1.4.6 + ``` -If you are unfamiliar with Terraform, check out this [basic introduction to Terraform](./intro-to-terraform.md). +If you are unfamiliar with Terraform, check out this [basic introduction](./intro-to-terraform.md). ### Install AWS CLI -The [AWS Command Line Interface (AWS CLI)](https://aws.amazon.com/cli/) is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts. Install the AWS command line tool by following the instructions found here: +The [AWS Command Line Interface (AWS CLI)](https://aws.amazon.com/cli/) is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts. -- [Install AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) +Install the AWS CLI by following the [AWS installation instructions](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). ### Install Go The [Go programming language](https://go.dev/dl/) is required to run [Terratest](https://terratest.gruntwork.io/), the unit test framework for Terraform. +Use Homebrew to install go: + +```bash +brew install golang +``` + ### Install GitHub CLI -The [GitHub CLI](https://cli.github.com/) is useful for automating certain operations for GitHub such as with GitHub actions. This is needed to run [check-github-actions-auth.sh](/bin/check-github-actions-auth.sh) +The [GitHub CLI](https://cli.github.com/) is useful for automating certain operations on GitHub, such as GitHub actions. For example, you need the Github CLI to run [check-github-actions-auth.sh](/bin/check-github-actions-auth.sh). + +Use Homebrew to install the GitHub CLI: ```bash brew install gh @@ -44,52 +53,46 @@ brew install gh ### Install linters -We have several optional utilities for running infrastructure linters locally. These are run as part of the CI pipeline, therefore, it is often simpler to test them locally first. +The following linters are run as part of the CI pipeline: * [Shellcheck](https://github.com/koalaman/shellcheck) * [actionlint](https://github.com/rhysd/actionlint) * [markdown-link-check](https://github.com/tcort/markdown-link-check) +To install and run them locally, run: + ```bash brew install shellcheck brew install actionlint make infra-lint ``` -## AWS Authentication - -In order for Terraform to authenticate with your accounts you will need to configure your AWS credentials using the AWS CLI or manually create your config and credentials file. If you need to manage multiple credentials or create named profiles for use with different environments you can add the `--profile` option. +### Authenticate with AWS + +To use Terraform with your AWS accounts, you must configure your AWS credentials. There are multiple ways to authenticate with AWS, but we recommend the following process: + +1. Use the AWS CLI command `aws configure --profile ` to create a separate profile for each AWS account. `aws configure` will store your credentials in `~/.aws/credentials` (Linux & Mac) or `%USERPROFILE%\.aws\credentials` (Windows). For example, to create a profile named `my-aws-account`, run: + ```bash + $ aws configure --profile my-aws-account + AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE + AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY + Default region name [None]: us-east-2 + Default output format [None]: json + ``` +2. Set the local environment variable `AWS_PROFILE` to the profile name. For example, to set the `AWS_PROFILE` environment variable to `my-aws-account`, run: + ```bash + export AWS_PROFILE=my-aws-account + ``` +3. (Optional) Use the [direnv](https://direnv.net/) tool to manage local environment variables. Instead of directly exporting environment variables on your machine, allow direnv to automatically set environment variables depending on the directory you are working in. +4. Verify access by running the following command. It should print out the profile name you set in Step 1. + ```bash + aws sts get-caller-identity + ``` + To see a more human-readable account alias instead of the account, run: + ```bash + aws iam list-account-aliases + ``` -There are multiple ways to authenticate, but we recommend creating a separate profile for your project in your AWS credentials file and setting your local environment variable `AWS_PROFILE` to the profile name. We recommend using [direnv](https://direnv.net/) to manage local environment variables. -**Credentials should be located in ~/.aws/credentials** (Linux & Mac) or **%USERPROFILE%\.aws\credentials** (Windows) - -### Examples - -```bash -$ aws configure -AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE -AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY -Default region name [None]: us-east-2 -Default output format [None]: json -``` - -**Using the above command will create a [default] profile.** - -```bash -$ aws configure --profile dev -AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE -AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY -Default region name [None]: us-east-2 -Default output format [None]: json -``` - -**Using the above command will create a [dev] profile.** - -Once you're done, verify access by running the following command to print out information about the AWS IAM user you authenticated as. - -```bash -aws sts get-caller-identity -``` ### References diff --git a/docs/infra/set-up-monitoring-alerts.md b/docs/infra/set-up-monitoring-alerts.md deleted file mode 100644 index 5dd08b145..000000000 --- a/docs/infra/set-up-monitoring-alerts.md +++ /dev/null @@ -1,29 +0,0 @@ -# Set up monitoring notifications - -## Overview - -The monitoring module defines metric-based alerting policies that provide awareness into issues with the cloud application. The module supports integration with external incident management tools like Splunk-On-Call or Pagerduty. It also supports email alerts. - -### Set up email alerts. - -1. Add the `email_alerts_subscription_list` variable to the monitoring module call in the service layer - -For example: -``` -module "monitoring" { - source = "../../modules/monitoring" - email_alerts_subscription_list = ["email1@email.com", "email2@email.com"] - ... -} -``` -2. Run `make infra-update-app-service APP_NAME= ENVIRONMENT=` to apply the changes to each environment. -When any of the alerts described by the module are triggered notification will be sent to all emails specified in the `email_alerts_subscription_list` - -### Set up External incident management service integration. - -1. Set setting `has_incident_management_service = true` in app-config/main.tf -2. Get the integration URL for the incident management service and store it in AWS SSM Parameter Store by running the following command for each environment: -``` -make infra-configure-monitoring-secrets APP_NAME= ENVIRONMENT= URL= -``` -3. Run `make infra-update-app-service APP_NAME= ENVIRONMENT=` to apply the changes to each environment. diff --git a/docs/infra/set-up-network-custom-domains.md b/docs/infra/set-up-network-custom-domains.md new file mode 100644 index 000000000..c211b9167 --- /dev/null +++ b/docs/infra/set-up-network-custom-domains.md @@ -0,0 +1,109 @@ +# Set up custom domains + +Follow these instructions for **each network** (you can have one or more in your project) in your project. If the network or an application does not need custom domains, skip to the bottom of this document. + +Production systems typically use custom domains to route internet traffic to their application services instead of AWS-generated hostnames for the load balancers or the CDN. + +The custom domain set up process will: + +* Create an [Amazon Route 53 hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) to manage DNS records for a domain and subdomains +* Create DNS A (address) records to route traffic from a custom domain to an application's load balancer + +## Prerequisites + +* You have registered custom domain(s) with a domain registrar (e.g. Namecheap, GoDaddy, Google Domains, etc.). +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You have [configured all application(s)](./set-up-app-config.md). +* You have [set up the networks](./set-up-network.md) that you want to add the custom domain to. +* You have [set up the application service](./set-up-app-service.md). + +## Instructions + +### 1. Set hosted zone in domain configuration + +The custom domain configuration is defined as a `domain_config` object in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf). A [hosted zone](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) represents a domain and all of its subdomains. For example, a hosted zone of `platform-test.navateam.com` includes `platform-test.navateam.com`, `cdn.platform-test.navateam.com`, `notifications.platform-test.navateam.com`, `foo.bar.platform-test.navateam.com`, etc. + +**For each network** you want to use a custom domain, in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf): + +1. Set the `hosted_zone` to match the custom domain (or a subdomain of the custom domain) that you registered. +2. Set `manage_dns` to `true`. + +### 2. Update the network layer to create the hosted zones + +**For each network** you that you added a custom domain to in the previous step, run the following command to create the hosted zone specified in the domain configuration: + +```bash +make infra-update-network NETWORK_NAME= +``` + +### 3. Delegate DNS requests to the newly created hosted zone + +You most likely registered your domain outside of this project. Using whichever service you used to register the domain name (e.g. Namecheap, GoDaddy, Google Domains, etc.), add a DNS NS (nameserver) record. Set the "name" equal to the `hosted_zone` and set the value equal to the list of hosted zone name servers that was created in the previous step. + +Output a list of servers by running: + +```bash +terraform -chdir=infra/networks output -json hosted_zone_name_servers +``` + +Your NS record might look something like this: + +**Name**: + +```text +platform-test.navateam.com +``` + +**Value**: (Note the periods after each of the server addresses) + +```text +ns-1431.awsdns-50.org. +ns-1643.awsdns-13.co.uk. +ns-687.awsdns-21.net. +ns-80.awsdns-10.com. +``` + +Verify that DNS requests are being served by the hosted zone nameservers by running the following command: + +```bash +nslookup -type=NS +``` + +### 4. Create DNS A (address) records to route traffic from the custom domain to the application's load balancer + +**For each application** in the network that should use the custom domain, perform the following. + +Within the `app-config` directory (e.g. `/infra//app-config`), each environment has its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. + +In each environment config file, define the `domain_name`. + +The `domain_name` must be either the same as the `hosted_zone` or a subdomain of the `hosted_zone`. For example, if your hosted zone is `platform-test.navateam.com`, then `platform-test.navateam.com` and `cdn.platform-test.navateam.com` are both valid values for `domain_name`. + +### 5. Update the application service + +**For each application and each environment** in the network that should use the custom domain, apply the changes by running the following command. Review the Terraform output carefully before typing "yes" to apply the changes. + +```bash +make infra-update-app-service APP_NAME= ENVIRONMENT= +``` + +`` must be the name of the application folder within the `/infra` folder. + +`` must be the name of the environment to update. + +## If a network does not need custom domains + +For each network that does not need custom domains, set the network's `domain_config` object in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) to the following: + +```hcl +domain_config = { + manage_dns = false + hosted_zone = "" + certificate_configs = {} +} +``` + +## If an application does not need custom domains + +For each application that does not need custom domains, in the application's `app-config/.tf` file (e.g. `/infra//app-config/.tf`), set `domain_name` to `""` (empty string). diff --git a/docs/infra/set-up-network-https.md b/docs/infra/set-up-network-https.md new file mode 100644 index 000000000..f96b649d0 --- /dev/null +++ b/docs/infra/set-up-network-https.md @@ -0,0 +1,72 @@ +# Set up HTTPS + +Follow these instructions for **each network** (you can have one or more in your project) in your project. If the network or an application does not need HTTPS, skip to the bottom of this document. + +To prevent man-in-the-middle attacks, production systems should use HTTPS rather than HTTP. + +The HTTPS set up process will: + +* Issue an SSL/TLS certificate using Amazon Certificate Manager (ACM) for each domain that should support HTTPS +* Associate the certificate with the application's load balancer, so that the load balancer can serve HTTPS requests + +## Prerequisites + +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. +* You have [set up custom domains](./set-up-network-custom-domains.md) and met all of those prerequisites. + * This is because SSL/TLS certificates must be properly configured for the specific domain to support establishing secure connections. + +## Instructions + +### 1. Set desired certificates in domain configuration + +**For each network** you want to configure, modify the network in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) to set the `certificate_configs` key. + +Set the `source` of the domain or subdomain to `issued`. + +### 2. Update the network layer to issue the certificates + +**For each network** you configured in the previous step, apply the changes by running the following command. Review the Terraform output carefully before typing "yes" to apply the changes. This will issue SSL/TLS certificates. + +```bash +make infra-update-network NETWORK_NAME= +``` + +Run the following command to check the status of a certificate (replace `` with the output from the previous command): + +```bash +aws acm describe-certificate --certificate-arn --query Certificate.Status +``` + +### 3. Update `enable_https = true` in `app-config` + +**For each application and environment** that should use HTTPS, perform the following. + +Within the `app-config` directory (e.g. `/infra//app-config`), each environment has its own config file named after the environment. For example, if the application has three environments `dev`, `staging`, and `prod`, it should have corresponding `dev.tf`, `staging.tf`, and `prod.tf` files. + +In each environment config file, set `enable_https` to `true`. This will attach the SSL/TLS certificate to the load balancer. + +You should have already set `domain_name` as part of [setting up custom domain names](/docs/infra/set-up-network-custom-domains.md). + +### 4. Attach certificate to load balancer + +**For each application and environment** that should use HTTPS, apply the changes from the previous step by running the following command. Review the Terraform output carefully before typing "yes" to apply the changes. + +```bash +make infra-update-app-service APP_NAME= ENVIRONMENT= +``` + +`` must be the name of the application folder within the `/infra` folder. + +`` must be the name of the environment to update. + +## If a network does not need HTTPS + +**⚠️ This is not advised** for any network containing a production environment. + +For each network that does not need custom domains, in [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf), set the network's `certificate_configs` to `{}` (empty hash). + +## If an application does not need HTTPS + +**⚠️ This is not advised** for an application deployed to a production environment. + +For each application that does not need HTTPS, in the application's `app-config/.tf` file (e.g. in `/infra//app-config/.tf`), set `enable_https` to `false`. \ No newline at end of file diff --git a/docs/infra/set-up-network.md b/docs/infra/set-up-network.md index b411c33bd..482f5f825 100644 --- a/docs/infra/set-up-network.md +++ b/docs/infra/set-up-network.md @@ -1,35 +1,33 @@ # Set up network -The network setup process will configure and deploy network resources needed by other modules. In particular, it will: +Follow these instructions for **each network** in your project (you can have one or more in your project). -1. Create a nondefault VPC -2. Create public subnets for publicly accessible resources such as the application load balancer, private subnets for the application service, and private subnets for the database. -3. Create VPC endpoints for the AWS services needed by ECS Fargate to fetch the container image and log to AWS CloudWatch. If your application has a database, it will also create VPC endpoints for the AWS services needed by the database layer and a security group to contain those VPC endpoints. +The network set up process will configure and deploy network resources needed for one network. In particular, it will: -## Requirements +* Create a nondefault VPC +* Create public subnets for publicly accessible resources such as the application load balancer, private subnets for the application service, and private subnets for the database +* Create VPC endpoints for the AWS services needed by ECS Fargate to fetch the container image and log to AWS CloudWatch. If your application has a database, it will also create VPC endpoints for the AWS services needed by the database layer and a security group to contain those VPC endpoints. -Before setting up the network you'll need to have: +## Prerequisites -1. [Set up the AWS account](./set-up-aws-account.md) -2. Optionally adjust the configuration for the networks you want to have on your project in the [project-config module](/infra/project-config/networks.tf). By default, there are three networks defined, one for each application environment. If you have multiple apps and want your applications in separate networks, you may want to give the networks differentiating names (e.g. "foo-dev", "foo-prod", "bar-dev", "bar-prod", instead of just "dev", "prod"). - 1. Optionally, [configure custom domains](/docs/infra/set-up-custom-domains.md). You can also come back to setting up custom domains at a later time. - 2. Optionally, [configure HTTPS support](/docs/infra/https-support.md). You can also come back to setting up HTTPS support at a later time. -3. [Configure the app](/infra/app/app-config/main.tf). - 1. Update `has_database` to `true` or `false` depending on whether or not your application has a database to integrate with. This setting determines whether or not to create VPC endpoints needed by the database layer. - 2. Update `has_external_non_aws_service` to `true` or `false` depending on whether or not your application makes calls to an external non-AWS service. This setting determines whether or not to create NAT gateways, which allows the service in the private subnet to make requests to the internet. - 3. Update `network_name` for your application environments. This mapping ensures that each network is configured appropriately based on the application(s) in that network (see `local.apps_in_network` in [/infra/networks/main.tf](/infra/networks/main.tf)) Failure to set the network name properly means that the network layer may not receive the correct application configurations for `has_database` and `has_external_non_aws_service`. +* You are [authenticated into the AWS account](./set-up-infrastructure-tools.md#authenticate-with-aws) you want to configure. +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You have [configured all application(s)](./set-up-app-config.md). +* You have [configured the project's networks](./set-up-networks.md). -## 1. Configure backend +## Instructions -To create the `tfbackend` file for the new network, run +### 1. Configure backend + +To create the `.tfbackend` file for the new network, run: ```bash make infra-configure-network NETWORK_NAME= ``` -## 2. Create network resources +### 2. Create network resources -Now run the following commands to create the resources. Review the terraform before confirming "yes" to apply the changes. +To create the resources, run the following command. Review the Terraform output carefully before typing "yes" to apply the changes. ```bash make infra-update-network NETWORK_NAME= @@ -37,4 +35,4 @@ make infra-update-network NETWORK_NAME= ## Updating the network -If you make changes to your application's configuration that impact the network (such as `has_database` and `has_external_non_aws_service`), make sure to update the network before you update or deploy subsequent infrastructure layers. +If you make changes to your application's configuration that impact the network (such as `has_database` and `has_external_non_aws_service`), update the network before you update subsequent infrastructure layers. diff --git a/docs/infra/set-up-networks.md b/docs/infra/set-up-networks.md new file mode 100644 index 000000000..d30528280 --- /dev/null +++ b/docs/infra/set-up-networks.md @@ -0,0 +1,27 @@ +# Set up networks + +The networks set up process will: + +* Configure project networks to be compatible with your project's environment mapping +* Configure all networks + +## Prerequisites + +* You have [set up the AWS account(s)](./set-up-aws-accounts.md). +* You have [configured all application(s)](./set-up-app-config.md). + +## Instructions + +### 1. Configure the project's networks + +Modify [`/infra/project-config/networks.tf`](/infra/project-config/networks.tf) so that the environments listed match what you decided on in the [set up AWS accounts](./set-up-aws-accounts.md) step. + +By default there are three networks defined, one for each application environment. You can add additional additional networks as desired. + +If you have multiple applications and want your applications in separate networks within the same AWS account, you may want to give the networks differentiating names (e.g. "foo-dev", "foo-prod", "bar-dev", "bar-prod", instead of just "dev", "prod"). + +Skip the `domain_config` config for now. These settings are configured later when [setting up custom domains](./set-up-network-custom-domains.md) and when [setting up HTTPS](./set-up-network-https.md). + +### 2. Set up each network + +For **each network** in `/infra/project-config/networks.tf`, [set up the network](./set-up-network.md). \ No newline at end of file diff --git a/docs/system-architecture.md b/docs/system-architecture.md index ba22b5f55..55400d4bb 100644 --- a/docs/system-architecture.md +++ b/docs/system-architecture.md @@ -15,7 +15,7 @@ This diagram shows the system architecture. [🔒 Make a copy of this Lucid temp * **GitHub** — Source code repository. Also responsible for Continuous Integration (CI) and Continuous Delivery (CD) workflows. GitHub Actions builds and deploys releases to an Amazon ECR registry that stores Docker container images for the application service. * **Incident Management Service** — Incident management service (e.g. PagerDuty or Splunk On-Call) for managing on-call schedules and paging engineers for urgent production issues. * **Service** — Amazon ECS service running the application. -* **Terraform Backend Bucket** — Amazon S3 bucket used to store terraform state files. +* **Terraform Backend Bucket** — Amazon S3 bucket used to store Terraform state files. * **Terraform Locks DynamoDB Table** — Amazon DynamoDB table used to manage concurrent access to terraform state files. * **VPC Endpoints** — VPC endpoints are used by the Database Role Manager to access Amazon Services without traffic leaving the VPC. * **VPC Network** — Amazon VPC network. diff --git a/infra/README.md b/infra/README.md index 63987dd0a..c861dd500 100644 --- a/infra/README.md +++ b/infra/README.md @@ -1,35 +1,35 @@ # Overview -This project practices infrastructure-as-code and uses the [Terraform framework](https://www.terraform.io). This directory contains the infrastructure code for this project, including infrastructure for all application resources. This terraform project uses the [AWS provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs). It is based on the [Nava platform infrastructure template](https://github.com/navapbc/template-infra). +This project practices infrastructure-as-code (IaC) and uses the [Terraform framework](https://www.terraform.io). This directory contains the infrastructure code for this project, including infrastructure for all application resources. This Terraform project uses the [AWS provider](https://registry.terraform.io/providers/hashicorp/aws/latest/docs). ## 📂 Directory structure -The structure for the infrastructure code looks like this: +The directory structure looks like this: ```text infra/ Infrastructure code - accounts/ [Root module] IaC and IAM resources - [app_name]/ Application directory: infrastructure for the main application + accounts/ [Root module] IaC and Identity Access Management (IAM) resources + [APP_NAME]/ Application directory: infrastructure for the [APP_NAME] application modules/ Reusable child modules - networks/ [Root module] Account level network config (shared across all apps, environments, and terraform workspaces) + networks/ [Root module] Account level network config (shared across all apps, environments, and Terraform workspaces) ``` Each application directory contains the following: ```text app-config/ Application-level configuration for the application resources (different config for different environments) - build-repository/ [Root module] Docker image repository for the application (shared across environments and terraform workspaces) + build-repository/ [Root module] Container image repository for the application (shared across environments and Terraform workspaces) database/ [Root module] Configuration for database (different config for different environments) service/ [Root module] Configuration for containers, such as load balancer, application service (different config for different environments) ``` -Details about terraform root modules and child modules are documented in [module-architecture](/docs/infra/module-architecture.md). +Details about Terraform root modules and child modules are documented in [module-architecture](/docs/infra/module-architecture.md). ## 🏗️ Project architecture ### 🧅 Infrastructure layers -The infrastructure template is designed to operate on different layers: +The infrastructure operates on different layers: - Account layer - Network layer @@ -45,42 +45,55 @@ This project has the following AWS environments: - `staging` - `prod` -The environments share the same root modules but will have different configurations. Backend configuration is saved as [`.tfbackend`](https://developer.hashicorp.com/terraform/language/settings/backends/configuration#file) files. Most `.tfbackend` files are named after the environment. For example, the `[app_name]/service` infrastructure resources for the `dev` environment are configured via `dev.s3.tfbackend`. Resources for a module that are shared across environments, such as the build-repository, use `shared.s3.tfbackend`. Resources that are shared across the entire account (e.g. /infra/accounts) use `..s3.tfbackend`. +The environments share the same root modules but have different configurations. Backend configuration is saved as [`.tfbackend`](https://developer.hashicorp.com/terraform/language/settings/backends/configuration#file) files. `.tfbackend` files are named as follows: + +* Most `.tfbackend` files are named after the environment. For example, resources in the `dev` environment for the `/service` module use `dev.s3.tfbackend`. +* Resources for modules that are shared across environments (e.g. `/build-repository`) use `shared.s3.tfbackend`. +* Resources that are shared across an entire AWS account (e.g. `/infra/accounts`) use `..s3.tfbackend`. ### 🔀 Project workflow -This project relies on Make targets in the [root Makefile](/Makefile), which in turn call shell scripts in [./bin](/bin). The shell scripts call `terraform` commands. Many of the shell scripts are also called by the [Github Actions CI/CD](/.github/workflows). +This project relies on Make targets in the [root Makefile](/Makefile), which in turn call scripts in [/bin](/bin). The scripts call Terraform commands. Many of the scripts are also called by the [Github Actions CI/CD](/.github/workflows). -Generally, you should use the Make targets or the underlying bin scripts, but you can call the underlying terraform commands if needed. See [making-infra-changes](/docs/infra/making-infra-changes.md) for more details. +Generally, you should use the Make targets or the underlying scripts, but, if needed, you can call the underlying Terraform commands. For more details, see [making infrastructure changes](/docs/infra/making-infra-changes.md). ## 💻 Development ### 1️⃣ First time initialization +#### Prerequisites + +* You have [installed](/README.md#installation) this template into an application that meets the [Application Requirements](/README.md#application-requirements). +#### Instructions -To set up this project for the first time (i.e., it has never been deployed to the target AWS account): +If this project has never been deployed to the target AWS account(s), complete the following steps: -1. [Install this template](/README.md#installation) into an application that meets the [Application Requirements](/README.md#application-requirements) -2. [Configure the project](/infra/project-config/main.tf) (These values will be used in subsequent infra setup steps to namespace resources and add infrastructure tags.) -3. [Set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md) -4. [Set up AWS account](/docs/infra/set-up-aws-account.md) -5. [Set up the virtual network (VPC)](/docs/infra/set-up-network.md) +1. [Configure the project](/infra/project-config/main.tf). +2. [Set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md). +3. [Set up AWS account(s)](/docs/infra/set-up-aws-accounts.md). +4. For each application: + 1. [Configure the application](/docs/infra/set-up-app-config.md). +5. [Set up network(s)](/docs/infra/set-up-networks.md). 6. For each application: - 1. [Set up application build repository](/docs/infra/set-up-app-build-repository.md) - 2. [Set up application database](/docs/infra/set-up-database.md) - 3. [Set up application environment](/docs/infra/set-up-app-env.md) - 4. [Configure environment variables and secrets](/docs/infra/environment-variables-and-secrets.md) - 5. [Set up background jobs](/docs/infra/background-jobs.md) + 1. [Configure environment variables and secrets](/docs/infra/set-up-environment-variables-and-secrets.md). + 2. [Set up application build repository](/docs/infra/set-up-app-build-repository.md). + 3. For each environment: + 1. [Set up application database](/docs/infra/set-up-app-database.md). + 2. [Set up application service](/docs/infra/set-up-app-service.md). + 3. (Optional) [Set up application monitoring alerts](/docs/infra/set-up-app-monitoring-alerts.md). + 4. (Optional) [Set up application background jobs](/docs/infra/background-jobs.md). +7. (Optional) [Set up custom domains](/docs/infra/set-up-network-custom-domains.md). +8. (Optional) [Set up HTTPS support](/docs/infra/set-up-network-https.md). ### 🆕 New developer To get set up as a new developer on a project that has already been deployed to the target AWS account: -1. [Set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md) -2. [Review how to make changes to infrastructure](/docs/infra/making-infra-changes.md) -3. (Optional) Set up a [terraform workspace](/docs/infra/intro-to-terraform-workspaces.md) +1. [Set up infrastructure developer tools](/docs/infra/set-up-infrastructure-tools.md). +2. [Review how to make changes to infrastructure](/docs/infra/making-infra-changes.md). +3. (Optional) [Set up a Terraform workspace](/docs/infra/intro-to-terraform-workspaces.md). ## 📇 Additional reading -Additional documentation can be found in the [documentation directory](/docs/infra). +Additional documentation is located in [documentation directory](/docs/infra). diff --git a/template-only-bin/download-and-install-app.sh b/template-only-bin/download-and-install-app.sh new file mode 100755 index 000000000..774db5a42 --- /dev/null +++ b/template-only-bin/download-and-install-app.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# ----------------------------------------------------------------------------- +# This script downloads and installs infrastructure for an application. +# Run this script in your project's root directory. +# +# Positional parameters: +# APP_NAME (required) - the name for the application, use kebab-case +# ----------------------------------------------------------------------------- +set -euo pipefail + +APP_NAME=$1 +CURRENT_VERSION=$(cat .template-version) + +# Enforce kebab-case +APP_NAME_KEBAB=$(echo "$APP_NAME" | tr "_" "-") + +echo "Cloning template-infra..." +git clone https://github.com/navapbc/template-infra.git + +echo "Switching to this project's current version of the template..." +cd template-infra +git checkout "$CURRENT_VERSION" >& /dev/null +cd - >& /dev/null + +# Install the app +curl "https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/install-app.sh" | bash -s -- "$APP_NAME_KEBAB" + +echo "Cleaning up template-infra folder..." +rm -fr template-infra + +echo "...Done." \ No newline at end of file diff --git a/template-only-bin/install-app.sh b/template-only-bin/install-app.sh new file mode 100644 index 000000000..ffc1939e1 --- /dev/null +++ b/template-only-bin/install-app.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +# ----------------------------------------------------------------------------- +# This script installs infrastructure for an application. +# It is called by other scripts. +# +# Positional parameters: +# APP_NAME (required) - the name for the application, use kebab-case +# DST_PREFIX (optional) - the directory that the application should be installed in +# Defaults to "" (the current directory). If directory is supplied, must contain a +# trailing slash. +# ----------------------------------------------------------------------------- +set -euo pipefail + +APP_NAME=$1 +DST_PREFIX=${2:-""} + +echo "Creating a terraform module for application: $APP_NAME..." +cp -r template-infra/infra/app "${DST_PREFIX}infra/$APP_NAME" + +# Helper to get the correct sed -i behavior for both GNU sed and BSD sed (installed by default on macOS) +# Hat tip: https://stackoverflow.com/a/38595160 +sedi () { + if sed --version >/dev/null 2>&1; then + sed -i -- "$@" + else + sed -i "" "$@" + fi +} +# Export the function so it can be used below +export -f sedi + +echo "Setting up CI/CD for application: $APP_NAME..." +cp template-infra/.github/workflows/cd-app.yml "${DST_PREFIX}.github/workflows/cd-$APP_NAME.yml" +# This regex will capture all instances of `app` that end in a space, a double quote, a forward slash, or a hyphen +# We do this to avoid accidentally replacing the keyword `app_name` +LC_ALL=C sedi "s/app\([\s\"\/\-]\)/$APP_NAME\1/g" "${DST_PREFIX}.github/workflows/cd-$APP_NAME.yml" + +cp template-infra/.github/workflows/ci-app-vulnerability-scans.yml "${DST_PREFIX}.github/workflows/ci-$APP_NAME-vulnerability-scans.yml" +LC_ALL=C sedi "s/app\([\s\"\/\-]\)/$APP_NAME\1/g" "${DST_PREFIX}.github/workflows/ci-$APP_NAME-vulnerability-scans.yml" diff --git a/template-only-bin/update-template.sh b/template-only-bin/update-template.sh index 2acf18b51..e783eae52 100755 --- a/template-only-bin/update-template.sh +++ b/template-only-bin/update-template.sh @@ -1,39 +1,237 @@ -#!/bin/bash +#!/usr/bin/env bash # ----------------------------------------------------------------------------- -# This script updates template-infra in your project. Run -# This script from your project's root directory. +# This script updates template-infra in your project. +# Run this script in your project's root directory. +# +# This script uses git to create patch files between the current HEAD and the +# TARGET_VERSION argument. For the main portion of the template, it sets the upstream +# github repo as a remote, creates a patch file, and applies it. The git-apply excludes +# template-only files. For applications, because they often do not retain the default +# `app` name, a different approach is needed. To create the patch for applications, the +# upstream repo is cloned into a sub-directory and `git diff --no-index` is used to +# compare differently-named directories. +# +# Usage: +# ./template-only-bin/update-template.sh # # Positional parameters: -# TARGET_VERSION (optional) – the version of template-infra to upgrade to. -# Defaults to main. +# APP_NAMES (required) – a comma-separated list (no spaces) of the apps in `/infra`. App +# names are expected to be hyphen-separated (i.e. kebab-case). +# Examples: `app`, `app,app2`, `my-app,your-app` +# +# TARGET_VERSION (optional) – the version of the template to install +# Defaults to main. Can be a branch, commit hash, or tag. +# +# TARGET_VERSION_TYPE (optional) – the type of TARGET_VERSION provided +# Defaults to branch. Can be: branch, commit, or tag. +# +# Examples: +# - To update a project with one application named `app` to `main` in the template repo: +# ./template-only-bin/update-template.sh app +# +# - To update a project with two applications to a specific commit: +# ./template-only-bin/update-template.sh app,app2 d42963d007e55cc37ef666019428b1d06a25cf71 commit +# +# - To update a project with three applications to a tag: +# ./template-only-bin/update-template.sh alpha,beta,gamma-three v0.8.0 tag # ----------------------------------------------------------------------------- set -euo pipefail -TARGET_VERSION=${1:-"main"} - +APP_NAMES=$1 +TARGET_VERSION=${2:-"main"} +TARGET_VERSION_TYPE=${3:-"branch"} CURRENT_VERSION=$(cat .template-version) +TARGET_VERSION_HASH="" -echo "Clone template-infra" -git clone https://github.com/navapbc/template-infra.git +echo "=====================================================================" +echo "Updating template-infra" +echo "=====================================================================" +echo "APP_NAMES=$APP_NAMES" +echo "CURRENT_VERSION=$CURRENT_VERSION" +echo "TARGET_VERSION=$TARGET_VERSION" +echo "TARGET_VERSION_TYPE=$TARGET_VERSION_TYPE" +echo -echo "Creating patch" -cd template-infra -git checkout "$TARGET_VERSION" +# Check: that the git repo is clean +if [ -n "$(git status --porcelain)" ]; then + echo "Error: Commit or stash all changes before proceeding. Exiting." + exit 1 +fi + +# Check: that $APP_NAMES is not an empty string +if [ -z "$APP_NAMES" ]; then + echo "Error: APP_NAMES cannot be empty." + echo " Please supply a comma-separated list of applications in /infra." + echo " Example: app" + echo " Example: app,app2" + echo "Exiting." + exit 1 +fi + +# Check: that all the apps exist +# Loop through the comma-separated list of apps +for APP_NAME in ${APP_NAMES//,/ } +do + if [ ! -d "infra/$APP_NAME" ]; then + echo "Error: infra/$APP_NAME does not exist. Exiting." + exit 1 + fi +done + +# Check: that TARGET_VERSION_TYPE is valid +case $TARGET_VERSION_TYPE in + "branch"|"commit"|"tag") + # Acceptable options, do nothing + ;; + *) + echo "Error: TARGET_VERSION_TYPE must be: branch, commit, or tag. Exiting." + exit 1 +esac -# Get version hash to update .template-version after patch is successful -TARGET_VERSION_HASH=$(git rev-parse HEAD) +echo "---------------------------------------------------------------------" +echo "1. Patching: main template" +echo "---------------------------------------------------------------------" +echo "Temporarily creating remote 'upstream-template-infra'..." +echo +git remote add upstream-template-infra https://github.com/navapbc/template-infra.git + +echo "Fetching from upstream remote..." +echo +git fetch upstream-template-infra >& /dev/null + +# Get target version hash +echo "Converting $TARGET_VERSION to hash..." +case $TARGET_VERSION_TYPE in + "branch") + TARGET_VERSION_HASH=$(git rev-parse "upstream-template-infra/$TARGET_VERSION") + ;; + "commit") + echo "No conversion needed." + TARGET_VERSION_HASH=$TARGET_VERSION + ;; + "tag") + TARGET_VERSION_HASH=$(git ls-remote --tags upstream-template-infra "$TARGET_VERSION" | cut -d$'\t' -f1) + ;; +esac +echo "TARGET_VERSION_HASH=$TARGET_VERSION_HASH" +echo # Note: Keep this list in sync with the files copied in install-template.sh -git diff "$CURRENT_VERSION" "$TARGET_VERSION" -- .github bin docs infra Makefile .dockleconfig .grype.yml .hadolint.yaml .trivyignore > update.patch -cd - +INCLUDE_PATHS=( + ".github" + "bin" + "docs" + "infra" + "Makefile" + ".dockleconfig" + ".grype.yml" + ".hadolint.yaml" + ".trivyignore" +) + +# Note: Exclude terraform deployment files, and CI/CD workflows as those are handled +# separately below. +EXCLUDE_PATHS=( + "':!*.terraform*'" + "':!*.tfbackend'" + "':!.github/workflows/'" +) + +# Exclude all applications +for APP_NAME in ${APP_NAMES//,/ } +do + EXCLUDE_PATHS+=("':!infra/$APP_NAME'") +done -echo "Applying patch" -# Note: Keep this list in sync with the removed files in install-template.sh -EXCLUDE_OPT="--exclude=.github/workflows/template-only-*" -git apply "$EXCLUDE_OPT" --allow-empty template-infra/update.patch +# Show the changes to be made +STAT_COMMAND="git --no-pager diff -R --stat $TARGET_VERSION_HASH -- ${INCLUDE_PATHS[*]} ${EXCLUDE_PATHS[*]}" +eval "$STAT_COMMAND" -echo "Saving new template version to .template-infra" +# Make the patch file +DIFF_COMMAND="git diff -R $TARGET_VERSION_HASH -- ${INCLUDE_PATHS[*]} ${EXCLUDE_PATHS[*]}" +eval "$DIFF_COMMAND > main-template.patch" + +# Apply the patch file +git apply --allow-empty main-template.patch + +echo +echo "---------------------------------------------------------------------" +echo "2. Preparing to patch application(s)" +echo "---------------------------------------------------------------------" +git clone https://github.com/navapbc/template-infra.git +cd template-infra +git checkout "$TARGET_VERSION_HASH" +cd - >& /dev/null + +# Patch each application +STEP_COUNT=3 +for APP_NAME in ${APP_NAMES//,/ } +do + echo + echo "---------------------------------------------------------------------" + echo "$STEP_COUNT. Patching application: $APP_NAME" + echo "---------------------------------------------------------------------" + # If the APP_NAME is not named `app`, then install a new app in template-infra to diff against + if [ "$APP_NAME" != "app" ]; then + curl "https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/install-app.sh" | bash -s -- "$APP_NAME" "template-infra/" + echo + fi + + # Copy the latest files into the working tree + cp -r "template-infra/infra/$APP_NAME" infra + + # Add any untracked files as intent-to-add + git add --intent-to-add "infra/$APP_NAME" + + # To create a git patch comparing a project's application: + # || true is necessary because this bash script includes `set -e` option, which will + # immediately exit for any non-zero exit codes. That's generally correct, but + # `git diff --no-index` will return 1 to indicate differences between the files. + git diff "infra/$APP_NAME" > "template-infra/$APP_NAME.patch" || true + + # Restore infra dir + git checkout "infra/$APP_NAME" + + # The stat version of the `git-apply` command`, used to output the changes to STDOUT + STAT_COMMAND="git --no-pager apply --stat --allow-empty template-infra/$APP_NAME.patch --exclude='*.tfbackend' --exclude='*.terraform*' --exclude='*.tfstate*'" + eval "$STAT_COMMAND" + + # Actually run the `git apply` command + git apply --allow-empty "template-infra/$APP_NAME.patch" --exclude="*.tfbackend" --exclude="*.terraform*" --exclude="*.tfstate*" + + # Increment step counter + STEP_COUNT=$((STEP_COUNT+1)) +done + +echo +echo "---------------------------------------------------------------------" +echo "$STEP_COUNT. Patching CI/CD" +echo "---------------------------------------------------------------------" +# This follows the same pattern as above +git diff --no-index --dst-prefix="" .github/workflows template-infra/.github/workflows > "template-infra/ci-$APP_NAME.patch" || true +STAT_COMMAND="git --no-pager apply --stat --allow-empty template-infra/ci-$APP_NAME.patch --exclude='.github/workflows/template-only*'" +eval "$STAT_COMMAND" +git apply --allow-empty "template-infra/ci-$APP_NAME.patch" --exclude=".github/workflows/template-only*" +STEP_COUNT=$((STEP_COUNT+1)) + +echo +echo "---------------------------------------------------------------------" +echo "$STEP_COUNT. Cleaning up" +echo "---------------------------------------------------------------------" +echo "Saving new template version to .template-infra..." echo "$TARGET_VERSION_HASH" > .template-version -echo "Clean up template-infra folder" -rm -fr template-infra +echo "Removing patch files..." +rm main-template.patch + +echo "Removing git remote..." +git remote rm upstream-template-infra + +echo "Cleaning up template-infra folder..." +rm -rf template-infra + +echo +echo "=====================================================================" +echo "Done." +echo "=====================================================================" +echo "Review all changes carefully using 'git diff' before committing" \ No newline at end of file diff --git a/template-only-docs/application-requirements.md b/template-only-docs/application-requirements.md index d3caace1a..8637b451d 100644 --- a/template-only-docs/application-requirements.md +++ b/template-only-docs/application-requirements.md @@ -22,13 +22,13 @@ If your application needs a database, it must also: ## Example Application -The infra template includes an example "hello, world" application that works with the template. The source code for this test application is at [app](/app). +This template includes an example "hello, world" application that works with the template. The source code for this test application is at [`/app`](/app). -A live demo of the test application is fully deployed by the repo, which is used for testing the infra template. Please check [that repo's README](https://github.com/navapbc/platform-test?tab=readme-ov-file#environment-urls) to locate a URL for seeing the live demo. +A live demo of the test application is fully deployed by the repo, which is used for testing this template. Please check [that repo's README](https://github.com/navapbc/platform-test?tab=readme-ov-file#environment-urls) to locate a URL for seeing the live demo. ## Template Applications -You can use the following template applications with the template infrastructure. Each of these includes a script to generate a working application that works with this infra template. +You can use the following template applications with this infrastructure template. Each of these includes a script to generate a working application that works with this template. * [template-application-nextjs](https://github.com/navapbc/template-application-nextjs) * [template-application-flask](https://github.com/navapbc/template-application-flask) diff --git a/template-only-docs/multiple-applications.md b/template-only-docs/multiple-applications.md new file mode 100644 index 000000000..ad3da1b07 --- /dev/null +++ b/template-only-docs/multiple-applications.md @@ -0,0 +1,38 @@ +# Multiple Applications + +You can use this template with multiple applications. By default, this template assumes your project has one application named `app`. However, it's straightforward to add additional applications. + +## Prerequisites + +* You have [installed](/README.md#installation) this template. + +## Instructions to add an additional application + +### 1. Ensure the application meets the Application Requirements + +Applications must meet [these requirements](/template-only-docs/application-requirements.md) to be used with this template. If you're using a [Platform application template](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), these requirements are already met. + +### 2. Add the application to the root directory + +Add the application's source code to a folder in the project's root folder (e.g. `/second-app`). + +⚠️ **Warning:** In general, it's best to use short, descriptive one-word names because some AWS resources have character limits. If you must use multiple words, use hyphens (not underscores) to separate each word. + +### 3. Add infrastructure support for the application + +To add the Terraform modules for the application, run: + +```bash +curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/download-and-install-app.sh | bash -s -- +``` + +`` must be the name of the application you chose in the previous step. + +This will add a new terraform module at `/infra/` and create the following CI/CD workflows: + +* `/.github/workflows/cd-.yml` +* `/.github/workflows/ci--vulnerability-scans.yml` + +### 4. Configure the application as usual + +Follow the per-application steps in [`/infra/README.md`](/infra/README.md) to complete setup. \ No newline at end of file diff --git a/template-only-docs/set-up-cd.md b/template-only-docs/set-up-cd.md index d94437717..d9d6785b7 100644 --- a/template-only-docs/set-up-cd.md +++ b/template-only-docs/set-up-cd.md @@ -1,6 +1,8 @@ # Set up CD -Once you have set up your application environments, you can enable continuous deployment in [cd-app.yml](../.github/workflows/cd-app.yml) by searching for `!!` and following the instructions to: +Once you have set up your application(s), you can enable continuous deployment. Each application should have a CD Github Actions workflow (e.g. `/.github/workflows/cd-`). + +In each `/.github/workflows/cd-`, search for `!!` and following the instructions to: 1. Update the `role-to-assume` with the GitHub actions ARN. 2. Uncomment the `on: push: ["main]` workflow trigger. This will trigger the deployment workflow on every merge to `main`. diff --git a/template-only-docs/set-up-ci.md b/template-only-docs/set-up-ci.md index 7264adfad..f8eccc13a 100644 --- a/template-only-docs/set-up-ci.md +++ b/template-only-docs/set-up-ci.md @@ -4,12 +4,10 @@ CI should automatically be set up once the CI files in `.github/workflows` are committed and pushed to the remote repository in GitHub. -Some checks are disabled until you've completed certain setup steps: +### Instructions -### After setting up the application environment +Some checks are disabled until you've completed all set up steps: -After [setting up the app environment](/docs/infra/set-up-app-env.md): - -- Uncomment the infra end-to-end tests by searching for `!!` in [ci-infra-service.yml](/.github/workflows/ci-infra-service.yml). You can verify that CI is running and passing by clicking into the Actions tab in GitHub. Note that this repo only contains CI for infra. Application CI (`ci-app.yml`) is included as part of the application templates. -- Uncomment the push trigger in [cd-app.yml](/.github/workflows/cd-app.yml) -- If you setup your AWS account in a different region than `us-east-1`, update the `aws-region` workflow settings to match your region. +1. Uncomment the infrastructure end-to-end tests by searching for `!!` in [`.github/workflows/ci-infra-service.yml`](/.github/workflows/ci-infra-service.yml). After uncommenting, verify that the CI is running and passing by clicking the Actions tab in GitHub. + * Note that this repo only contains CI for the infrastracture. If you're using one of the [Platform application templates](https://github.com/navapbc/platform?tab=readme-ov-file#platform-templates), then the application CI workflow (`/.github/workflows/ci-app.yml`) is already included. Otherwise, you'll need to create one. +2. If you setup your AWS account in a region other than `us-east-1`, update the `aws-region` workflow settings in [`/.github/workflows/check-infra-auth.yml`](/.github/workflows/check-infra-auth.yml) to match your region. diff --git a/template-only-docs/template-development-workflow.md b/template-only-docs/template-development-workflow.md index a5ddbbc01..ef80b81ae 100644 --- a/template-only-docs/template-development-workflow.md +++ b/template-only-docs/template-development-workflow.md @@ -4,7 +4,7 @@ This is the workflow for developers making changes to the infrastructure templat ## Prerequisites -For most infrastructure changes, you will need an environment to work with. Since template-infra is a template and not a live project, it doesn't have any long-lived environments. Thus, you should develop and test your infrastructure changes using the `dev` environment on one of the following test repos: +For most infrastructure changes, you will need an environment to work with. Since this repo is a template and not a live project, it doesn't have any long-lived environments. Thus, you should develop and test your infrastructure changes using the `dev` environment on one of the following test repos: * [platform-test](https://github.com/navapbc/platform-test) – Test project that uses [template-infra](https://github.com/navapbc/template-infra) and the [example app](https://github.com/navapbc/template-infra/tree/main/app) that comes with the template. This is the default project we use for development and testing infrastructure changes. * [platform-test-flask](https://github.com/navapbc/platform-test-flask) - Test project that uses [template-infra](https://github.com/navapbc/template-infra) and [template-application-flask](https://github.com/navapbc/template-application-flask) @@ -31,7 +31,7 @@ On the platform test repo, you'll do the following: 4. Create a pull request 5. Iterate until all CI checks pass on your PR and you’ve also done additional testing that you need to validate. *Do not merge the PR.* -### 2. Create a pull request on infra template repo +### 2. Create a pull request on the `template-infra` repo 1. Once you've completed development and testing, create a pull request on the [template-infra](https://github.com/navapbc/template-infra) repo with the same changes you made on the platform test repo. 2. In the "Testing" section of the PR description, link to the PR on the platform test repo as evidence of the testing you did to verify your changes. @@ -39,7 +39,7 @@ On the platform test repo, you'll do the following: ### 3. Push changes to platform test repos -In most cases, after you merge changes to the infra template, the changes will be automatically pushed to the various platform test repos. However, the following changes aren't automatically propagated to the platform test repos: +In most cases, after you merge changes to the `template-infra` repo, the changes will be automatically pushed to the various platform test repos. However, the following changes aren't automatically propagated to the platform test repos: 1. Changes to [this list of files in update-template.sh](https://github.com/navapbc/template-infra/blob/main/template-only-bin/update-template.sh#L17-L28) 2. Deletions of template files diff --git a/template-only-docs/update-template.md b/template-only-docs/update-template.md new file mode 100644 index 000000000..e1f0743b4 --- /dev/null +++ b/template-only-docs/update-template.md @@ -0,0 +1,46 @@ +# Updates + +## Prerequisites + +* The [update script](/template-only-bin/update-template.sh) assumes that your project is version-controlled using `git`. The script will edit your project files, but it will not run `git commit`. After running the script, use `git diff` to review all changes carefully. + +## Instructions + +To update your project to a newer version of this template, run the following command in your project's root directory: + +```bash +curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- +``` + +`` is a required argument. It must be a comma-separated list (no spaces) of the apps in `/infra`. App names must be hyphen-separated (i.e. kebab-case). Examples: `app`, `app,app2`, `my-app,your-app`. + +⚠️ **Warning:** Read the [release notes](https://github.com/navapbc/template-infra/releases) in case there are breaking changes you need to address manually. + +### Specifying a branch, a commit, or a tag + +By default, the update script will apply changes from the `main` branch of this template repo. If you want to update to a different branch, a specific commit, or a specific tag (e.g. a release tag), run this instead: + +```bash +curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- +``` + +`` should be the version of the template to install. This can be a branch, commit hash, or tag. + +`` should be the type of `` provided. Defaults to `branch`. This can be: `branch`, `commit`, or `tag`. + +### Examples + +If your project has one application named `app` and you want to update it to the `main` branch of this template repo, run: +```bash +curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- app +``` + +If your project has two applications named `app, app2` and you want to update to the commit `d42963d007e55cc37ef666019428b1d06a25cf71`, run: +```bash +curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- app,app2 d42963d007e55cc37ef666019428b1d06a25cf71 commit +``` + +If your project has three applications named `foo,bar,baz` and you want to update to the `v.0.8.0` release tag, run: +```bash +curl https://raw.githubusercontent.com/navapbc/template-infra/main/template-only-bin/update-template.sh | bash -s -- foo,bar,baz v0.8.0 tag +```