Cloud Run Sidecar - Admin Server not returning on the ready endpoint to Cloud Run

[john-ramsey]

Environment

• PostgreSQL version: 14.12
• PostgREST version: 12.2.2
• Operating system: x86_64-pc-linux-gnu, official Docker image

Description of issue

The admin server does not appear to report back to GCP cloud run that the service is ready upon

Expected Behavior:

• Admin server returns 200 OK

Actual Behavior:

• It looks like PostgREST is starting up fine but fails to report to cloud run. Has anyone else had this issue?

Reproduction

1. Start a cloud run service with PostgREST as a sidecar container
2. Have the main container startup dependant on PostgREST
3. Ensure that PostgREST has a startup probe (GCP will require this)
4. PostgREST will fail to report back to Cloud Run

[steve-chavez]

Can you try with https://github.com/PostgREST/postgrest/releases/tag/v12.2.0 or https://github.com/PostgREST/postgrest/releases/tag/v12.2.1?

Just to check if it's a problem with the latest version.

[john-ramsey]

I tried both tags and neither worked for me. Were you able to recreate?

[steve-chavez]

No, I cannot reproduce. Is it possible to isolate this from cloud run?

[john-ramsey]

Just to clarify you were not able to reproduce in Cloud Run or locally? I locally I am unable to reproduce outside of cloud run which has me thinking it is a result of how Cloud Run is routing sidecar health checks.

[wolfgangwalther]

What do you have server-host set to? Maybe you need to change that somehow?

[wolfgangwalther]

Were you able to get this to run / figure out what the problem was?

[john-ramsey]

Unfortunately this isn't something that we have prioritized. No answer as of yet as we are operating without the probe.

[thomasjungblut]

also running this into now, any hints?

[wolfgangwalther]

Same question as above, what did you set server-host to?

[thomasjungblut]

in the health probe request? why would you need to set that?

…

On Sun, 25 Aug 2024, 17:04 Wolfgang Walther, ***@***.***> wrote: Same question as above, what did you set server-host to? — Reply to this email directly, view it on GitHub <#3665 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGVYMGOJOHVG5EKAUJTJS3ZTHXBBAVCNFSM6AAAAABLA4DYCKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANBUGMZTMMA> . You are receiving this because you commented.Message ID: ***@***.***>

[thomasjungblut]

It's not a sidecar btw, it's the sole running container on cloud run.

It might be a limitation of cloudrun, which requires the exact same port (?): https://cloud.google.com/run/docs/troubleshooting#container-failed-to-start

You can also not configure more than one container port either:

ports should contain 0 or 1 port. (field: service.spec.template.spec.containers[0].ports)

It's a bit odd, because as you mentioned this works in k8s/knative pretty well.

[thomasjungblut]

Just for completeness sake, here's the full cloud run yaml:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: postgrest
  generation: 24
  labels:
    cloud.googleapis.com/location: europe-west4
  annotations:
    run.googleapis.com/ingress: all
    run.googleapis.com/ingress-status: all
spec:
  template:
    metadata:
      labels:
        run.googleapis.com/startupProbeType: Custom
      annotations:
        run.googleapis.com/client-name: cloud-console
        run.googleapis.com/cloudsql-instances: 'XXX:europe-west4:XXX-master'
        autoscaling.knative.dev/maxScale: '5'
        run.googleapis.com/startup-cpu-boost: 'false'
    spec:
      containerConcurrency: 100
      timeoutSeconds: 30
      serviceAccountName: [email protected]
      containers:
      - name: postgrest-1
        image: 'gcr.io/XXX/postgrest:v12.2.3'
        ports:
        - name: http1
          containerPort: 3000
        env:
        - name: PGRST_SERVER_HOST
          value: 0.0.0.0
        - name: PGRST_ADMIN_SERVER_PORT
          value: '3001'
        - name: PGRST_DB_PRE_CONFIG
          value: postgrest.pre_config
        - name: PGRST_JWT_SECRET_IS_BASE64
          value: 'true'
        - name: PGRST_DB_URI
          valueFrom:
            secretKeyRef:
              key: latest
              name: XXX-master-postgrest-url
        - name: PGRST_JWT_SECRET
          valueFrom:
            secretKeyRef:
              key: '1'
              name: XXX-master-postgrest-hmac
        resources:
          limits:
            cpu: 1000m
            memory: 128Mi
        startupProbe:
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 5
          failureThreshold: 5
          httpGet:
            path: ready
            port: 3001
  traffic:
  - percent: 100
    latestRevision: true

[wolfgangwalther]

If you run a startup probe only, then you don't need it to be super fast. So you could just query the root endpoint on port 3000 instead and don't use the admin server for that.

[thomasjungblut]

I need the server to return a proper result, otherwise my application will get a 500 error when it queries a table during the startup where the schema hasn't been loaded yet. Note that the cloud function has to spin up, get the database connection, load its caches and then serve traffic. It's quick and mitigated by client retries already, it just makes the availability look pretty poor.

I reckon I can also create a surrogate table that is ordinally at the very bottom of the list being loaded and use that as the startup probe path instead.

[wolfgangwalther]

I reckon I can also create a surrogate table that is ordinally at the very bottom of the list being loaded and use that as the startup probe path instead.

The schema cache is loaded at once, not sequentially. So it doesn't matter whether that's "at the end of the list" - there is no such list or end.

Instead of a table, you can also use a simple RPC function which returns void or so.