terraform module for dataform, with best practices baked in.

---

Built with:

This repository provides a comprehensive approach to setting up and managing Dataform on Google Cloud Platform (GCP) using Terraform. It leverages the power of Terraform to automate the provisioning of infrastructure and resources required for Dataform, ensuring consistency, repeatability, and ease of management.

contents

• dataform-terraform
  • overview
  • requirements
  • structure
  • terraform
  • best
  • resources
  • authors

---

1. Create a Google Cloud Project:
   • Navigate to the Google Cloud Console.
   • Create a project - Create a Project.
   • Enable billing for your project - Enable Billing.
2. Set Up a Git Repository:
   • Create a Git repository (e.g., on GitHub, Bitbucket) to store your Dataform code. This repository will be linked to your Dataform repository on GCP.
3. Generate SSH Keys:
   • Generate a public-private SSH key pair that you'll use to connect your Dataform repository to the Git repository: Generate SSH Keys.
4. Install Terraform: Download and install Terraform on your local machine: Install Terraform.
5. Install Google Cloud SDK: Install the Google Cloud SDK to manage your GCP resources: Install Google Cloud SDK.
6. Authenticate with Google Cloud: Authenticate your Google Cloud account using the Google Cloud SDK: Authenticate with Google Cloud.

---

.
├── gcs-trigger.tf       # Terraform configuration for Eventarc triggers.
├── backend.tf          # Terraform backend configuration.
├── main.tf             # Main Terraform configuration file.
├── workflows            # Directory containing Dataform workflow YAML files.
│   └── dataform.yaml    # Example Dataform workflow configuration.
├── variables.tf         # Terraform variables file.
└── .gitignore           # Files and directories to ignore in Git.

---

1. Configure Terraform
   • Copy one of the examples, being sure to use your own workflows/configs in the ./configs folder.
   • Configure your Terraform backend in a backend.tf if required.
2. Initialise Terraform
   • Run terraform init to initialise Terraform and download the necessary providers.
3. Plan Terraform Changes
   • Run terraform plan to review the infrastructure changes that Terraform will apply.
4. Apply Terraform Changes:
   • Run terraform apply to apply the changes to your GCP environment.
5. Update your SSH Key:
   • Add the SSH Key you Generated in place of the placeholder Secret Terraform Created.

requirements

Name	Version
terraform	>= 1.7.4
google	>= 6.11.2, < 7.0.0
google-beta	>= 6.11.2, < 7.0.0
random	>= 3.1.0, < 4.0.0
random	>= 3.1.0, < 4.0.0

providers

Name	Version
google	>= 6.11.2, < 7.0.0
google-beta	>= 6.11.2, < 7.0.0
http	>= 3.1.0, < 4.0.0
random	>= 3.1.0, < 4.0.0

modules

Name	Source	Version
dataform-sa	github.com/GoogleCloudPlatform/cloud-foundation-fabric//modules/iam-service-account	v36.0.1
dataform-ssh-key	github.com/GoogleCloudPlatform/cloud-foundation-fabric//modules/secret-manager	v36.0.1
project	github.com/GoogleCloudPlatform/cloud-foundation-fabric//modules/project	v36.0.1

resources

Name	Type
google-beta_google_dataform_repository.dataform_respository	resource
google-beta_google_dataform_repository_release_config.release_config	resource
google-beta_google_dataform_repository_workflow_config.workflow_config	resource
random_uuid.check	resource
google_client_config.default	data source
http_http.bitbucket	data source
http_http.github	data source
http_http.workspace	data source

inputs

Name	Description	Type	Default	Required
base_dir	The base directory for the dataform project	string	"workflows"	no
default_branch_name	The default branch name for the repository	string	"main"	no
labels	labels to apply to all resources	map(string)	{ "iac": "terraform" }	no
name	The name of the dataform repository	string	"dataform"	no
project_id	The project id	string	n/a	yes
region	The region where the project will be created	string	n/a	yes
repository_url	The repository url for dataform	string	n/a	yes
vcs	The vcs of choice - can only be 'gihub' or 'bitbucket'	string	"github"	no

outputs

Name	Description
http	n/a

---

• You must have a basic understanding of Terraform, Dataform, and Git to use this repository.
• The provided Terraform code is a template and may need adjustments to meet your specific requirements.

---

• Terraform Documentation
• Terraform Registry
• Google Cloud BigQuery
• Google Cloud Storage
• Google Cloud Eventarc
• Google Cloud Workflows
• Google Cloud Secret Manager

---

Brought to you by the incredible minds of Emile Hofsink and Benjamin Western.