Update google CFT module "terraform-google-bigquery"

[anik147]

TL;DR

Latest google provider version "v5.42.0" made resource type "google_bigquery_dataset_iam_member" non-authoritative which fixes the conflict with "google_bigquery_dataset_access" resource type. When adding member for same IAM role using "google_bigquery_dataset_iam_member" and "google_bigquery_dataset_access" resource type used to remove authorize views added in the dataset.
Can you please help us bring this changes to google CFT module "terraform-google-bigquery" so we don't face the above issue when adding IAM role using CFT module input block "access" and "google_bigquery_dataset_iam_member" resource type.

Terraform Resources

https://registry.terraform.io/modules/terraform-google-modules/bigquery/google/latest

Detailed design

No response

Additional information

No response

[sherintky]

Hi Team,

Please release new version which will support minimum google provider version as v5.42.0 (current provider version is version = ">= 5.3, < 7"). When adding same IAM role using CFT module input block "access" and "google_bigquery_dataset_iam_member" resource type, we are facing some conflict, authorize views added to dataset is getting removed. This issue is solved in google provider version as v5.42.0. Hence we would request you to update module terraform-google-bigquery minimum provider version to v5.42.0.

[bharathkkb]

Hi our constraint of >= 5.3, < 7 allows for the use of v5.42.0. You can explicitly pin to that in your root module if needed. Attached an example below. We generally only bump minimum versions when introducing new functionality that requires a minimum provider version.

module "bigquery" {
  source  = "terraform-google-modules/bigquery/google"
  version = "~> 8.1"

  dataset_id                  = "foo"
  dataset_name                = "foo"
}

terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
      version = "5.42"
    }
  }
}

terraform version
Terraform v1.5.7
on darwin_arm64
+ provider registry.terraform.io/hashicorp/google v5.42.0

[sherintky]

Hi Team,

We tried running the terraform plan & apply using google provider version v5.42 and above version. We are seeing IAM roles added using "access" and "google_bigquery_dataset_iam_member" blocks are still conflicting , roles are removing in first build and getting added in next build. This is continuing in every build.

please see the below terraform code and terraform plan for your reference.

#main.tf file
module "bigquery" {
source = "terraform-google-modules/bigquery/google"
version = "~> 8.1"
for_each = var.psvalues1
dataset_id = each.value["dataset_id"]
dataset_name = each.value["dataset_name"]
description = each.value["description"]
project_id = var.project_id
default_table_expiration_ms = each.value["default_table_expiration_ms"]
location = each.value["location"]
access = each.value["access"]
dataset_labels = each.value["dataset_labels"]
tables = each.value["tables"]
external_tables = each.value["external_tables"]
views = each.value["views"]
encryption_key = each.value["encryption_key"]
}

resource "google_bigquery_dataset_iam_member" "newtestdemo_member_sa_1" {
project = var.project_id
dataset_id = "newtestdemo"
role = "roles/bigquery.dataOwner"
member ="serviceAccount:[email protected]"
}

resource "google_bigquery_dataset_iam_member" "newtestdemo_member_sa_2" {
project = var.project_id
dataset_id = "newtestdemo"
role = "roles/bigquery.dataViewer"
member ="serviceAccount:[email protected]"
}

resource "google_bigquery_dataset_iam_member" "newtestdemo_member_user_1" {
project = var.project_id
dataset_id = "newtestdemo"
role = "roles/bigquery.dataOwner"
member ="user:[email protected]"
}

resource "google_bigquery_dataset_iam_member" "newtestdemo_member_group_1" {
project = var.project_id
dataset_id = "newtestdemo"
role = "roles/bigquery.dataViewer"
member ="group:[email protected]"
}

#variable file

variable "psvalues1" {
type = map(object({
dataset_id = string
dataset_name = string
description = string
default_table_expiration_ms = number
location = string
access = any
dataset_labels = map(string)
tables = any
external_tables = any
views = any
encryption_key = string
})
)
}

#tfvar file
psvalues1 = {
testdemo = {
dataset_id = "newtestdemo"
dataset_name = "newtestdemo"
description = ""
default_table_expiration_ms = "259200000"

       location          ="US"
   #access = []
   access = [
              {
                 role          = "OWNER"
                 user_by_email = "[email protected]"
              },
	  { 
	    role          = "READER"
                group_by_email = "[email protected]"
              },
	]
       dataset_labels = {}
       tables = []
       external_tables = []
       views = []
       encryption_key = null
  },

}

Terraform plan first time

module.bigquery["testdemo"].google_bigquery_dataset.main will be updated in-place

~ resource "google_bigquery_dataset" "main" {
id = "projects/test-project/datasets/newtestdemo"
# (20 unchanged attributes hidden)

  - access {
      - role           = "OWNER" -> null
      - user_by_email  = "[email protected]" -> null
        # (4 unchanged attributes hidden)
    }
  - access {
      - role           = "OWNER" -> null
      - user_by_email  = "[email protected]" -> null
        # (4 unchanged attributes hidden)
    }
  - access {
      - role           = "OWNER" -> null
      - user_by_email  = "[email protected]" -> null
        # (4 unchanged attributes hidden)
    }
  - access {
      - role           = "READER" -> null
      - user_by_email  = "[email protected]" -> null
        # (4 unchanged attributes hidden)
    }
  - access {
      - group_by_email = "[email protected]" -> null
      - role           = "READER" -> null
        # (4 unchanged attributes hidden)
    }
  - access {
      - group_by_email = "[email protected]" -> null
      - role           = "READER" -> null
        # (4 unchanged attributes hidden)
    }
  + access {
      + role           = "OWNER"
      + user_by_email  = "[email protected]"
        # (3 unchanged attributes hidden)
    }
  + access {
      + group_by_email = "[email protected]"
      + role           = "READER"
        # (3 unchanged attributes hidden)
    }
}

################################

Terraform plan second time

Terraform will perform the following actions:

google_bigquery_dataset_iam_member.newtestdemo_member_group_1 will be created

• resource "google_bigquery_dataset_iam_member" "newtestdemo_member_group_1" {
  • dataset_id = "newtestdemo"
  • etag = (known after apply)
  • id = (known after apply)
  • member = "group:[email protected]"
  • project = "test-project"
  • role = "roles/bigquery.dataViewer"
    }

google_bigquery_dataset_iam_member.newtestdemo_member_sa_1 will be created

• resource "google_bigquery_dataset_iam_member" "newtestdemo_member_sa_1" {
  • dataset_id = "newtestdemo"
  • etag = (known after apply)
  • id = (known after apply)
  • member = "serviceAccount:[email protected]"
  • project = "test-project"
  • role = "roles/bigquery.dataOwner"
    }

google_bigquery_dataset_iam_member.newtestdemo_member_sa_2 will be created

• resource "google_bigquery_dataset_iam_member" "newtestdemo_member_sa_2" {
  • dataset_id = "newtestdemo"
  • etag = (known after apply)
  • id = (known after apply)
  • member = "serviceAccount:[email protected]"
  • project = "test-project"
  • role = "roles/bigquery.dataViewer"
    }

google_bigquery_dataset_iam_member.newtestdemo_member_user_1 will be created

• resource "google_bigquery_dataset_iam_member" "newtestdemo_member_user_1" {
  • dataset_id = "newtestdemo"
  • etag = (known after apply)
  • id = (known after apply)
  • member = "user:[email protected]"
  • project = "test-project"
  • role = "roles/bigquery.dataOwner"
    }

[sherintky]

attached the terraform provider details