Remove Google Cloud Function for transferring data to BiqQuery

[chaecramb]

The functionality of this Google Cloud Function has been integrated directly into the Search API repo via two rake tasks alphagov/search-api-v2#351. These are scheduled via cron tasks: alphagov/govuk-helm-charts#2825

This commit removes the function itself and related Terraform scheduling.

[kevindew]

This looks good and the plan looks good 👍

I am expecting that there is probably more in this repo that can be removed off the back of this. First thing that comes to mind is the function itself: https://github.com/alphagov/search-v2-infrastructure/tree/remove-user-events-runs/terraform/environment/files/vertex_events_push ?

I'm not sure if there are other bits or comments that might be out of date too?

[kevindew]

I think these bits might be removable:

search-v2-infrastructure/terraform/environment/events_ingestion.tf

Lines 320 to 364 in 3c43d36

	# bucket for vertex bq -> vertex datastore function .zip
	resource "google_storage_bucket" "import_user_events_vertex_function" {
	name = "${var.gcp_project_id}_import_user_events_vertex"
	location = var.gcp_region
	}
	# zipped import_user_events_vertex function into bucket
	resource "google_storage_bucket_object" "import_user_events_vertex_function_zipped" {
	name = "import_user_events_vertex_function_${data.archive_file.import_user_events_vertex_function.output_md5}.zip"
	bucket = google_storage_bucket.import_user_events_vertex_function.name
	source = data.archive_file.import_user_events_vertex_function.output_path
	}
	# archive .py and requirements.txt for import_user_events_vertex to zip
	data "archive_file" "import_user_events_vertex_function" {
	type = "zip"
	source_dir = "${path.module}/files/vertex_events_push/"
	output_path = "${path.module}/files/vertex_events_push.zip"
	}
	# gen 2 function for transferring from bq - vertex events schema to vertex engine
	resource "google_cloudfunctions2_function" "import_user_events_vertex" {
	name = "import_user_events_vertex"
	description = "function that will trigger daily to transfer of ga4 events data in vertex schema in bq to vertex"
	location = var.gcp_region
	build_config {
	entry_point = "import_user_events_vertex"
	runtime = "python311"
	source {
	storage_source {
	bucket = google_storage_bucket.import_user_events_vertex_function.name
	object = google_storage_bucket_object.import_user_events_vertex_function_zipped.name
	}
	}
	}
	service_config {
	max_instance_count = 5
	ingress_settings = "ALLOW_ALL"
	service_account_email = google_service_account.analytics_events_pipeline.email
	environment_variables = {
	PROJECT_NAME = var.gcp_project_id
	}
	}
	}

[kevindew]

I'm a bit unclear on these roles too:

search-v2-infrastructure/terraform/environment/events_ingestion.tf

Lines 300 to 318 in 3c43d36

	# custom role for writing vertex analytics data to vertex datastore
	resource "google_project_iam_custom_role" "vertex_upload" {
	role_id = "vertex_upload"
	title = "bq-write-vertex-permissions"
	description = "Write data to vertex datastore from bq"
	permissions = [
	"discoveryengine.userEvents.import",
	"discoveryengine.userEvents.create"
	]
	}
	# binding ga write role to ga write service account
	resource "google_project_iam_binding" "vertex_datastore_write" {
	role = google_project_iam_custom_role.vertex_upload.id
	project = var.gcp_project_id
	members = [
	google_service_account.analytics_events_pipeline.member
	]
	}

not sure if they're something that is used or can be deleted

[chaecramb]

@kevindew yes, looks to me like these sections can also be removed. I've updated the PR accordingly.

[kevindew]

Thanks Chae - we've also got the function itself - the whole directory of https://github.com/alphagov/search-v2-infrastructure/tree/main/terraform/environment/files/vertex_events_push.

I'm going to have a scan see if there's anything else I can see that seems redundant.

[kevindew]

I've had a scan and I think it's just the function code that needs to be removed further.

[chaecramb]

@kevindew Ah yes, I didn't actually realise that was defined here, but it makes sense. I've removed it.

[kevindew]

Super, looks good to me