Firestore transaction hangs for > 1 minute without logs

[sjudd]

Environment details

• OS type and version: Google Cloud Run
• Python version: 3.11.7
• pip version: 23.3.1
• google-cloud-firestore version: 2.14.0

Steps to reproduce

I'm a little unclear. The issue seems flaky. It seems related to having a lot of concurrent requests, but the requests are for different documents (albiet in the same table).

The table in question is a listing of individual files. I'm using a on_object_finalized firestore to function to look for an existing object in the table using something like:

        query_ref = files_ref.where(filter=FieldFilter("bucket", "==", bucket)).where(
            filter=FieldFilter("path", "==", path)
        )
        docs = transaction.get(query_ref)

Then if one isn't found, I create a new entry:

   new_doc_ref = files_ref.document()
    data = {
        "path": path,
        "bucket": bucket,
   }
   transaction.create(new_doc_ref, data)

Overall concurrency is quite low, something like 7-9 requests in a second. I've seen it happen with less.

Code example

See above

Stack traces

Sadly no logs appear. The function runs until it hits a 60s timeout (it usually finishes in 300-500ms when warm, up to 1.5 seconds when cold). Then the function exists. A log line is printed immediately before the transaction starts. But a log line immediately after the transaction is not printed.

Maybe this is related to #850 and/or #358?

[garysmall]

Seeing the same issue with cloud run and concurrent requests. Causes the whole instance to be unusable until it is spun down. Each following requests also returns a 504.

[sjudd]

firebase/firebase-functions-python#181 has more details, a repro and a workaround from someone else.