[BUG] state.apply broken with minion 3007.1 when missing python dependencies

[ptitdoc]

Description
When running salt-master 3007.1, some clients in 3007.1 are not working properly when running state.apply.

I'm trying to understand why since several weeks/months and looking through several bugs related to instabilities with salt 3006.X/3007 which may or may not be related.
#66288
#66282

After looking to running salt-call state.apply directly on the minion because it hangs on the master, I end up with this error:

  File "/usr/lib/python3.12/site-packages/salt/modules/state.py", line 420, in _check_queue
    queue = __salt__["config.option"]("state_queue", False)
            ~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/salt/loader/context.py", line 85, in __getitem__
    return self.value()[item]
           ~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3.12/site-packages/salt/loader/lazy.py", line 350, in __getitem__
    super().__getitem__(item)  # try to get the item from the dictionary
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/salt/utils/lazy.py", line 105, in __getitem__
    raise KeyError(key)
KeyError: 'config.option'

The error comes from module config that cannot loads because it fails to load a bunch of (useless?) dependencies, which seems to be critical as module.config is a core feature:

[ERROR   ] Failed to import module config, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/salt/loader/lazy.py", line 770, in _load_module
    spec.loader.exec_module(mod)
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/usr/lib/python3.12/site-packages/salt/modules/config.py", line 22, in <module>
    import salt.utils.cloud
  File "/usr/lib/python3.12/site-packages/salt/utils/cloud.py", line 82, in <module>
    import pkg_resources  # pylint: disable=3rd-party-module-not-gated
    ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkg_resources/__init__.py", line 3291, in <module>
    @_call_aside
     ^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkg_resources/__init__.py", line 3266, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.12/site-packages/pkg_resources/__init__.py", line 3304, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkg_resources/__init__.py", line 600, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.12/site-packages/pkg_resources/__init__.py", line 937, in require
    needed = self.resolve(parse_requirements(requirements))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkg_resources/__init__.py", line 798, in resolve
    dist = self._resolve_dist(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pkg_resources/__init__.py", line 839, in _resolve_dist
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'rpm-vercmp' distribution was not found and is required by salt

The problem is traced down to there:

salt/salt/modules/config.py

Line 20 in 246d066

try:

The module properly initialise with HAS_CLOUD set to False if there is an ImportError, so we should not care about the salt.utils.cloud dependencies. But it fails with a DistributionNotFound instead and just crash...

If I remove all the try block and set HAS_CLOUD=False, everything run properly.

Please not that while installing missing dependencies, I also found other exceptions types than ImportError and DistributionNotFound