Podman machine rootless connection fails due to `storage.conf` permissions in FCOS image

[Honny1]

Workeround:

podman machine ssh "<machine-name>" "sudo chmod 755 /etc/containers" && podman machine "<machine-name>" stop && podman machine start "<machine-name>"

Issue Description

After upgrading to podman 6 podman machine start, the rootless user-level podman.socket inside the VM immediately enters a crash loop and systemd kills it with trigger-limit-hit. The root cause is that /etc/containers/storage.conf inside the FCOS image has incorrect permissions, causing every podman.service invocation to fail with open /etc/containers/storage.conf: permission denied.

Steps to reproduce the issue

Steps to reproduce the issue

1. podman machine init test
2. podman machine start test
3. podman -c test info

Describe the results you received

Host:

$ podman machine start test
Starting machine "test"
Warning: The machine being started is not set as your default Podman connection.
As such, Podman commands may not work correctly.
Set the default Podman connection to this machine? [y/N] n
Default system connection will remain unchanged

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:

	podman machine set --rootful test2

WARN[0013] API socket failed ping test
API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.

Machine "test2" started successfully
$ podman -c test info
OS: darwin/arm64
buildOrigin: pkginstaller
provider: libkrun
version: 6.0.0

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to connect: dial tcp 127.0.0.1:52768: connect: connection refused

Inside the VM:

core@localhost:~$ systemctl --user status podman.socket
× podman.socket - Podman API Socket
     Loaded: loaded (/usr/lib/systemd/user/podman.socket; enabled; preset: disabled)
     Active: failed (Result: trigger-limit-hit) since Thu 2026-06-25 15:24:30 CEST; 21min ago
   Duration: 751ms
 Invocation: b178aff8be6f47a9b8a3b8123e05a9c6
   Triggers: ● podman.service
       Docs: man:podman-system-service(1)
     Listen: /run/user/501/podman/podman.sock (Stream)

Jun 25 15:24:30 localhost.localdomain systemd[1693]: Listening on podman.socket - Podman API Socket.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.socket: Trigger limit hit, refusing further activation.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.socket: Failed with result 'trigger-limit-hit'.

core@localhost:~$ journalctl --user -u podman.socket -u podman.service --no-pager -l
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Listening on podman.socket - Podman API Socket.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[1897]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[1912]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[1952]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[1964]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[1976]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[1994]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[2001]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[2007]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[2023]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[2030]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[2037]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[2045]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[2052]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[2059]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[2066]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[2073]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[2080]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[2087]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Starting podman.service - Podman API Service...
Jun 25 15:24:30 localhost.localdomain systemd[1693]: Started podman.service - Podman API Service.
Jun 25 15:24:30 localhost.localdomain podman[2094]: Failed to obtain podman configuration: open /etc/containers/storage.conf: permission denied
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.service: Failed with result 'exit-code'.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.socket: Trigger limit hit, refusing further activation.
Jun 25 15:24:30 localhost.localdomain systemd[1693]: podman.socket: Failed with result 'trigger-limit-hit'.

Describe the results you expected

podman machine start should result in a working machine where podman info connects successfully. /etc/containers/storage.conf should be world-readable inside the FCOS image.

podman info output

OS: darwin/arm64
buildOrigin: pkginstaller
provider: libkrun
version: 6.0.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Honny1]

PTAL @Luap99 @podman-container-tools/podman-maintainers

[Luap99]

/etc/containers is over mounted from the host ~/.config/containers directory?

So do you have a storage.conf file there? What permissions does it have?

[Honny1]

/etc/containers is over mounted from the host ~/.config/containers directory?

Yes

$ mount | grep /etc/containers
9ddeaea5d41a791d2cc3bcc676b9645fee0e on /etc/containers type virtiofs (rw,relatime,context=system_u:object_r:nfs_t:s0)

So do you have a storage.conf file there?

No, I would expect that installer will create that.

What permissions does it have?

ls -la /etc/ | grep containers
drwx------   7 root root    224 Jun 25 15:22 containers

I was able to work around that with: (inside podman machine)

sudo chmod 755 /etc/containers

[Honny1]

I tested the podman machine that uses AppleHV, and it works. Is possible that it is related to: #29030 (comment) ?

[Luap99]

No, I would expect that installer will create that.

No installer will create it this file is not needed. My question was more do you have that file in your host config dir and it somehow has restricted perms but seems like the problem is the directory not just the file

What created ~/.config/containers/ on the host? it should be created with 755 by default. If it is not created with that we need to fix that. If a user manually set these perms we likely need to rethink the approach when that happens?

I tested the podman machine that uses AppleHV, and it works. Is possible that it is related to: #29030 (comment) ?

it could be

[Honny1]

What created ~/.config/containers/ on the host? it should be created with 755 by default. If it is not created with that we need to fix that. If a user manually set these perms we likely need to rethink the approach when that happens?

I think installer. I didn't do any edits to the host. I changed the permissions to that directory inside the podman machine.

I tested the podman machine that uses AppleHV, and it works. Is possible that it is related to: #29030 (comment) ?

it could be

I will try to build an installer with updated libkrun.

[Honny1]

I built a new installer with libkrun 1.2.2, and it seems that it works. WDYT? @Luap99

[Honny1]

Well, I removed ~/.config/containers and then installed podman 6.0 and did init, etc., and it works as expected.

This is before (worked with libkrun 1.2.2):

$ stat ~/.config/containers
16777231 33746712 drwx------ 7 jrodak staff 0 224 "Jun 25 16:36:49 2026" "Jun 25 16:58:57 2026" "Jun 25 16:58:57 2026" "Dec  2 16:47:28 2024" 4096 0 0 /Users/jrodak/.config/containers

$ ls -la ~/.config/ | grep containers
drwx------@  7 jrodak  staff   224 Jun 25 16:58 containers

This is after:

$ stat ~/.config/containers
16777231 165328240 drwxr-xr-x 5 jrodak staff 0 160 "Jun 25 17:15:55 2026" "Jun 25 17:17:23 2026" "Jun 25 17:17:23 2026" "Jun 25 17:15:55 2026" 4096 0 0 /Users/jrodak/.config/containers

$ ls -la ~/.config/ | grep containers
drwxr-xr-x@  5 jrodak  staff   160 Jun 25 17:17 containers

[Luap99]

Ok good to know, so the problem may only exists for older installations at least, hard to say how far back this goes.

FWIW I did check with my mac:

% stat ~/.config/containers
16777229 73848 drwxr-xr-x 5 paul staff 0 160 "Oct 29 13:35:08 2024" "Jan  8 11:45:35 2026" "Jan  8 11:45:35 2026" "Jun  7 15:24:17 2024" 4096 0 0 /Users/paul/.config/containers

which shows the directory being older than yours, but I guess I could also have done a chmod at some point so that does not say much.

If this works with the latest libkrun update then we have at least a fix for the older installations if this affects many people

[Honny1]

Okay, lets sumarize that. The krunkit 1.2.1 can set incorrect permissions on /etc/containers/ inside the VM, preventing rootless podman from reading storage.conf and causing the user-level podman.socket to crash loop until systemd kills it with trigger-limit-hit. This can happend when host has unexpecrted permission on ~/.config/containers. It should be fixed in krunkit 1.2.2; workaround for existing machines: podman machine ssh <name> "sudo chmod 755 /etc/containers" then restart. Or remove ~/.config/containers but it will destroy your configuration.