PREFECT_HOME not used when serving flow

[kknuut-aavamobile]

Bug summary

When I try to serve flow from systemd procress, Prefect3 does not use PREFECT_HOME to setup profile location. Instead it forcibly uses /root/.prefect/profiles.toml

This used to work. Only change is prefect2 updated to prefect3 on server machine and this affected client machine.

Previously working version prefect=2.14.11
New current version prefect=3.2.0
both Python 3.10.15

Profile:

active = "to_machine1"

[profiles.default]
PREFECT_API_URL = "http://127.0.0.1:89/api"
PREFECT_API_TLS_INSECURE_SKIP_VERIFY = "1"

[profiles.to_machine1]
PREFECT_LOCAL_STORAGE_PATH = "/srv/prefect2/storage"
PREFECT_API_URL = "http://machine1:89/api"

Flow code:

import os
print(os.environ)
if(__name__ == "__main__"):
  print("Starting Deployment...")
  dep = ResultsUpdateToxx.to_deployment( .... ) 
  serve(dep)

Command to start on systemd service

 PREFECT_HOME='/home/username/.prefect' /srv/python-kernels/envs/shopfloor/bin/prefect profile ls
 PREFECT_HOME='/home/username/.prefect' /srv/python-kernels/envs/shopfloor/bin/python3 "Flow.py"

It can also be reproduced with sudo:

sudo PREFECT_HOME='/home/username/.prefect' /srv/python-kernels/envs/shopfloor/bin/python3 "Flow.py"

Output:

┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Available Profiles: ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│             default │
│       * to_machine1│
└─────────────────────┘
   * active profile
10:28:12.176 | INFO    | prefect - Starting temporary server on http://127.0.0.1:8308
See https://docs.prefect.io/3.0/manage/self-host#self-host-a-prefect-server for more information on running a dedicated Prefect server.
environ({'PREFECT_HOME': '/home/username/.prefect', 'LC_ADDRESS': '   ......<snipped>
Starting Deployment...
Traceback (most recent call last):
  File "/srv/python-kernels/envs/shopfloor/lib/python3.10/site-packages/prefect/deployments/runner.py", line 327, in _create
    deployment_id = await client.create_deployment(**create_payload)
  File "/srv/python-kernels/envs/shopfloor/lib/python3.10/site-packages/prefect/client/orchestration/_deployments/client.py", line 685, in create_deployment
    response = await self.request(
  File "/srv/python-kernels/envs/shopfloor/lib/python3.10/site-packages/prefect/client/orchestration/base.py", line 53, in request
    return await self._client.send(request)
  File "/srv/python-kernels/envs/shopfloor/lib/python3.10/site-packages/prefect/client/base.py", line 354, in send
    response.raise_for_status()
  File "/srv/python-kernels/envs/shopfloor/lib/python3.10/site-packages/prefect/client/base.py", line 162, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url 'http://127.0.0.1:8308/api/deployments/'

Debugging & info:

• Python sees PREFECT_HOME enviroment variable correctly inside Flow.py.
• Prefect is installed to micromamba environment. Enviroment binaries are called directly without trying to enable enviroment for root.
• Running prefect profile ls for root shows empty. No profiles.toml file for root.
• Trying PREFECT_SERVER_ALLOW_EPHEMERAL_MODE=False just resulted in failure without local server.
• Copying /home/username/.prefect/profiles.toml to /root/.prefect/ fixes problem and flow connects to correct server.

Version info

Version:             3.2.0
API version:         0.8.4
Python version:      3.10.15
Git commit:          c8986ede
Built:               Fri, Feb 7, 2025 6:02 PM
OS/Arch:             linux/x86_64
Profile:             to_oulxs002
Server type:         server
Pydantic version:    2.10.3

Additional context

No response

[kknuut-aavamobile]

Above issue was replicated also on another client machine.

Possibly related issues:
#16665
#15815

[zzstoatzz]

hi @kknuut-aavamobile - thanks for the issue! I think I know what's going on here. If you set PREFECT_PROFILES_PATH to your specific profiles.toml, does that work as expected? I think the issue is that setting PREFECT_PROFILES_PATH is no longer being inferred by the set PREFECT_HOME value.

I am thinking we should change the profile source for the Settings object to fix this

[zzstoatzz]

hi @kknuut-aavamobile - can you try installing from main and see if this is resolved for you?

pip install git+https://github.com/prefecthq/prefect.git

[zzstoatzz]

this is now released as prefect==3.2.1