Use of `GoogleAuth.getClient()` via `gax-nodejs` via `@googlemaps/addressvalidation` in AWS Lambda causes slow GCE metadata fetch attempt

[jstephenson]

Please make sure you have searched for information in the following guides.

• Search the issues already opened: https://github.com/GoogleCloudPlatform/google-cloud-node/issues
• Search StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform+node.js
• Check our Troubleshooting guide: https://github.com/googleapis/google-cloud-node/blob/main/docs/troubleshooting.md
• Check our FAQ: https://github.com/googleapis/google-cloud-node/blob/main/docs/faq.md
• Check our libraries HOW-TO: https://github.com/googleapis/gax-nodejs/blob/main/client-libraries.md
• Check out our authentication guide: https://github.com/googleapis/google-auth-library-nodejs
• Check out handwritten samples for many of our APIs: https://github.com/GoogleCloudPlatform/nodejs-docs-samples

A screenshot that you have tested with "Try this API".

API does not seem to be available in the explorer

Link to the code that reproduces this issue. A link to a public Github Repository or gist with a minimal reproduction.

https://gist.github.com/jstephenson/d8a44a2d28b73923796952aa8a02d081

A step-by-step description of how to reproduce the issue, based on the linked reproduction.

1. Execute the above sample in an AWS Lambda function
   • This should be reproducible in any environment which does not provide project ID in one of the pre-metadata fashions encoded here: https://github.com/googleapis/google-auth-library-nodejs/blob/a511e562783402468093d70e77e73762fa95ae79/src/auth/googleauth.ts#L299-L303

A clear and concise description of what the bug is, and what you expected to happen.

I have attempted a trivial usage of the Google Maps Platform's Address Validation API from an AWS Lambda function today. This service uses an API key for authentication.

It takes ~15 seconds for execution to resume at the API call site. Upon inspection—primarily via NODE_DEBUG=http—I can see three failed attempts to connect to a GCP metadata endpoint at 169.254.169.254 which each have a five second timeout.

As documented I am using the apiKey auth strategy for the Address Validation API. Looking at the code, it seems that GoogleAuth's getClient() method kicks off a call stack which ends up making a call into the gcp-metadata package's project() function in my use-case. This then in turn attempts to reach a link-local metadata server, and this then becomes slow.

To validate that this is indeed the issue I have set the environment variable GOOGLE_CLOUD_PROJECT on my AWS Lambda function and verified that no attempts are then made to reach a metadata service.

I can't find any documentation which suggests I am doing anything wrong, so I'm raising this here in the belief that this represents a bug. Sorry if I'm wrong.

A clear and concise description WHY you expect this behavior, i.e., was it a recent change, there is documentation that points to this behavior, etc. **

The Address Validation API uses API key authentication and this appears to be supported by the SDK

[feywind]

@jstephenson Thanks for taking the time to fill out a thorough report. I thought that looked like one of the private IP ranges, and it looks like this specific address is commonly used for AWS, Azure, and GCP for metadata services. I wonder if it's being routed to the AWS metadata service, and the GCP client libraries get confused about the response.

[jstephenson]

Yes indeed: this is an address in the IANA link-local block, and this metadata service approach is common across public clouds in various compute contexts. I suspect this particular address is common simply because of its memorability.

As above, this is a timeout in my case rather than a problem interpreting a response. AWS Lambda does not have an IP-addressable metadata service.

The core of my report is that the logic highlighted above assumes both tenancy on GCP and availability of its metadata service. Neither of these are valid assumptions in the presented use case and cause undesirable and confusing behaviour.

The use of gcp-metadata's detectGCPResidency() function seems as though it could be a part of a solution here.

[sofisl]

We recently had an issue with GCE and older version of auth that were causing Incorrect Metadata Flavor errors. It's possible that the errors started here. To confirm, could you update auth to the latest version, npm i @google-auth-library@latest, and retry at your earliest convenience?

[jstephenson]

I'm already using the latest version:

❯ pnpm why google-auth-library
(...)

dependencies:
@googlemaps/addressvalidation 2.4.0
└─┬ google-gax 4.4.1
  └── google-auth-library 9.15.1

[sofisl]

I see. I think I'm going to transfer this issue over to gcp-metadata then, and label it as a feature request to better support AWS.