4-projects step 15 production apply of both business units - after common apply - fails with API rate limit on concurrent operations quota - fix: retry up to 60 min later

[obriensystems]

step 15 Step #4 - "tf apply": module.env.module.firewall_rules.google_compute_network_firewall_policy_rule.rules["1000"]: Creation complete after 32s [id=projects/prj-p-bu2sample-peering-snb4/global/firewallPolicies/fp-p-peering-project-firewalls/rules/1000]
Step #4 - "tf apply": module.env.google_compute_network_firewall_policy_association.vpc_association: Creating...
Step #4 - "tf apply": module.env.google_compute_network_firewall_policy_association.vpc_association: Creation complete after 9s [id=projects/prj-p-bu2sample-peering-snb4/global/firewallPolicies/fp-p-peering-project-firewalls/associations/fp-p-peering-project-firewalls-vpc-p-peering-base]
Step #4 - "tf apply":
Step #4 - "tf apply": Error: Error adding network peering: googleapi: Error 403: Rate Limit Exceeded
Step #4 - "tf apply": Details:
Step #4 - "tf apply": [
Step #4 - "tf apply": {
Step #4 - "tf apply": "@type": "type.googleapis.com/google.rpc.ErrorInfo",
Step #4 - "tf apply": "domain": "compute.googleapis.com",
Step #4 - "tf apply": "metadatas": {
Step #4 - "tf apply": "containerId": "17819846901",
Step #4 - "tf apply": "containerType": "PROJECT",
Step #4 - "tf apply": "location": "global"
Step #4 - "tf apply": },
Step #4 - "tf apply": "reason": "CONCURRENT_OPERATIONS_QUOTA_EXCEEDED"
Step #4 - "tf apply": },
Step #4 - "tf apply": {
Step #4 - "tf apply": "@type": "type.googleapis.com/google.rpc.Help",
Step #4 - "tf apply": "links": [
Step #4 - "tf apply": {
Step #4 - "tf apply": "description": "Concurrent operations quota documentation.",
Step #4 - "tf apply": "url": "https://cloud.google.com/compute/operations-quota#concurrent_operation_limits"
Step #4 - "tf apply": }
Step #4 - "tf apply": ]
Step #4 - "tf apply": },
Step #4 - "tf apply": {
Step #4 - "tf apply": "@type": "type.googleapis.com/google.rpc.LocalizedMessage",
Step #4 - "tf apply": "locale": "en-US",
Step #4 - "tf apply": "message": "Quota on concurrent operations exceeded for project 17819846901 in global."
Step #4 - "tf apply": }
Step #4 - "tf apply": ]
Step #4 - "tf apply": , rateLimitExceeded
Step #4 - "tf apply":
Step #4 - "tf apply": with module.env.module.peering.google_compute_network_peering.peer_network_peering,
Step #4 - "tf apply": on .terraform/modules/env.peering/modules/network-peering/main.tf line 50, in resource "google_compute_network_peering" "peer_network_peering":
Step #4 - "tf apply": 50: resource "google_compute_network_peering" "peer_network_peering" {
Step #4 - "tf apply":
Finished Step #4 - "tf apply"
ERROR
ERROR: build step 4 "us-central1-docker.pkg.dev/prj-b-cicd-82vv/tf-runners/terraform:v1" failed: step exited with non-zero status: 1

part of #360

Step #4 - "tf apply": module.env.module.firewall_rules.google_compute_network_firewall_policy_rule.rules["1000"]: Creation complete after 32s [id=projects/prj-p-bu2sample-peering-snb4/global/firewallPolicies/fp-p-peering-project-firewalls/rules/1000]
Step #4 - "tf apply": module.env.google_compute_network_firewall_policy_association.vpc_association: Creating...
Step #4 - "tf apply": module.env.google_compute_network_firewall_policy_association.vpc_association: Creation complete after 9s [id=projects/prj-p-bu2sample-peering-snb4/global/firewallPolicies/fp-p-peering-project-firewalls/associations/fp-p-peering-project-firewalls-vpc-p-peering-base]
Step #4 - "tf apply": 
Step #4 - "tf apply": Error: Error adding network peering: googleapi: Error 403: Rate Limit Exceeded
Step #4 - "tf apply": Details:
Step #4 - "tf apply": [
Step #4 - "tf apply":   {
Step #4 - "tf apply":     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
Step #4 - "tf apply":     "domain": "compute.googleapis.com",
Step #4 - "tf apply":     "metadatas": {
Step #4 - "tf apply":       "containerId": "17819846901",
Step #4 - "tf apply":       "containerType": "PROJECT",
Step #4 - "tf apply":       "location": "global"
Step #4 - "tf apply":     },
Step #4 - "tf apply":     "reason": "CONCURRENT_OPERATIONS_QUOTA_EXCEEDED"
Step #4 - "tf apply":   },
Step #4 - "tf apply":   {
Step #4 - "tf apply":     "@type": "type.googleapis.com/google.rpc.Help",
Step #4 - "tf apply":     "links": [
Step #4 - "tf apply":       {
Step #4 - "tf apply":         "description": "Concurrent operations quota documentation.",
Step #4 - "tf apply":         "url": "https://cloud.google.com/compute/operations-quota#concurrent_operation_limits"
Step #4 - "tf apply":       }
Step #4 - "tf apply":     ]
Step #4 - "tf apply":   },
Step #4 - "tf apply":   {
Step #4 - "tf apply":     "@type": "type.googleapis.com/google.rpc.LocalizedMessage",
Step #4 - "tf apply":     "locale": "en-US",
Step #4 - "tf apply":     "message": "Quota on concurrent operations exceeded for project 17819846901 in global."
Step #4 - "tf apply":   }
Step #4 - "tf apply": ]
Step #4 - "tf apply": , rateLimitExceeded
Step #4 - "tf apply": 
Step #4 - "tf apply":   with module.env.module.peering.google_compute_network_peering.peer_network_peering,
Step #4 - "tf apply":   on .terraform/modules/env.peering/modules/network-peering/main.tf line 50, in resource "google_compute_network_peering" "peer_network_peering":
Step #4 - "tf apply":   50: resource "google_compute_network_peering" "peer_network_peering" {
Step #4 - "tf apply": 
Finished Step #4 - "tf apply"
ERROR
ERROR: build step 4 "us-central1-docker.pkg.dev/prj-b-cicd-82vv/tf-runners/terraform:v1" failed: step exited with non-zero status: 1

[obriensystems]

#391
retry fixes the API quota CONCURRENT_OPERATIONS_QUOTA_EXCEEDED issue - wait 60 min - now we see all shared VPCs for business units

[obriensystems]

doc https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/wiki/DevOps#concurrent-operations-quota-exceeded-during-terraform-apply

[obriensystems]

reference: terraform-google-modules/terraform-example-foundation#1154

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days