Health checks created via blueprints show up, but don't work without intervention

[DavidFuchs]

Describe the Bug

Due to the number of services I run in my homelab, I use Docker labels to automate the creation of public resources in Pangolin.

I recently started creating health checks for all my services, using Docker labels (blueprints). Unfortunately, while the health checks show up in the configuration for the public resources, they never take effect until I manually modify the health check in the UI.

It is enough to simply toggle the health check off and on again, but ideally it would work without having to interact with it at all.

Environment

• OS Type & Version: Ubuntu 24.04.04 & Docker 29.6.2
• Pangolin Version: 1.21.0
• Edition (Community or Enterprise): Enterprise (Personal Use)
• Gerbil Version: 1.4.3
• Traefik Version: 3.7
• Newt Version: 1.15
• Client Version: N/A

To Reproduce

Add health check labels to a Docker container in your compose.yml file. I'm using Immich as an example below, but it doesn't make a difference which container it is. The only labels that matter in the example below are the healthcheck labels.

When starting the compose stack, Pangolin picks up the labels and applies the blueprint. A new public resource is registered, and everything works except for the health check. The health check remains permanently 'Unknown'.

It can be resolved manually with these steps:

1. Open the Pangolin UI portal.
2. Go to Public Resources and press Edit on the resource.
3. Click the 'HTTP Settings' tab.
4. Click the 'Unknown' button under the 'Health Check' label.
5. Toggle 'Enable Health Checks' off, and then on again. Press Done.
6. Press Save Settings.

The health check will now show 'Healthy' and use the settings it got from the Docker labels.

While it's nice that the health check can be manually fixed like this, it means I still need to do manual updates any time I create a new public resource.

    labels:
      - "pangolin.public-resources.immich.name=Immich"
      - "pangolin.public-resources.immich.protocol=http"
      - "pangolin.public-resources.immich.full-domain=immich.mydomain.com"
      - "pangolin.public-resources.immich.targets[0].method=http"
      - "pangolin.public-resources.immich.targets[0].hostname=immich"
      - "pangolin.public-resources.immich.targets[0].port=2283"
      - "pangolin.public-resources.immich.targets[0].healthcheck.hostname=immich"
      - "pangolin.public-resources.immich.targets[0].healthcheck.port=2283"
      - "pangolin.public-resources.immich.targets[0].healthcheck.scheme=http"
      - "pangolin.public-resources.immich.targets[0].healthcheck.method=GET"
      - "pangolin.public-resources.immich.targets[0].healthcheck.path=/"

Expected Behavior

I would expect that after the blueprint is applied, the health check is created and starts checking the health of the service, without me having to do make any changes in the UI.

[AstralDestiny]

Should have immich at /api/server/ping for the healthcheck over / but Will check in a bit.

[DavidFuchs]

Thanks for the updated health check endpoint path!

Interestingly, when I updated the stack to use that new path, it went straight to Unhealthy in Pangolin, and then a few moments later went to Healthy. Point is, it didn't get stuck at Unknown.

I brought down the Immich compose stack and deleted the public resource for it in Pangolin. Then I brought up the compose stack - the health check totally came up properly again, starting out at Unhealthy and then went to Healthy a few moments later. It didn't go to Unknown this time. I put the path back to / to see if it had something to do with the path, and it still comes up properly (doesn't initialize as 'Unknown').

Now, perhaps something is being cached here even though I deleted the public resource, because I fully expected I'd have to toggle the health check off/on again. I did restart Pangolin a little while back... I wonder if I just ended up in a weird state that's resolved itself? Curious to hear if you're able to reproduce.

[AstralDestiny]

Yeah will check once I'm home

[AstralDestiny]

Sorry for delays though. (Also sorry if it reads like I'm responding like I'm in discord as I am responding within discord)

[AstralDestiny]

Mmm can't replicate comes up as healthy right off the bat, Check your blueprints area when you bring it up with the labels, to see if it's correct or parsed properly.

[DavidFuchs]

Ok, I see the issue now. The services I'm specifically having trouble with don't specify a path for the health check.

I've found through repeated trial and error that the health check fails to initialize properly unless all three of hostname, port and path are specified, even though path isn't documented as a required field. It appears I can omit scheme and method, but not path. I assumed that path would default to /. And maybe it does... but either way not setting the path leaves the health check in an Unknown state for me. I was specifying a path for my Immich example above, though I admit I probably added it after creating the resource without it in an Unknown health check state.

Another finding - if I specify just the health check hostname, or just the port but not both, the entire public service will never be picked up as a blueprint. If the public resource already exists, and I break the health check labels by removing hostname or port, the public service will simply stop being managed by blueprints, and no errors will appear.

Either way, I have a path forward now, thanks!

[AstralDestiny]

I try to find where the healthcheck wants to report for if you go to / you can cause app level issues for a healthcheck or it might rate limit you as it's constantly serving a single ip the same redirect and it keeps asking for the path.

Make sure hostname matches what the service from newt to that service will use.

[oschwartz10612]

Wil have a default to / in the next release thanks for finding this issue