[BUG] x509.certificate_managed remote sign fails after enabling x509_v2

[joachimtingvold]

Description
Hit bug #65474 (same error message) while using the old/deprecated x509 module. Switched to x509_v2 on the CA/PKI minion, and the "client" minion. Now it fails with the following error:

client1.foobar.com:
----------
          ID: ssl_cert_nginx
    Function: x509.certificate_managed
        Name: /etc/pki/client1_nginx.crt
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "check_cmd determined the state failed"
              check_cmd determined the state failed
     Started: 07:22:28.280979
    Duration: 30881.262 ms
     Changes:

The check_cmd worked fine with x509 v1. Trying to remove check_cmd while v2 is enabled, it still fails;

client1.foobar.com:
----------
          ID: ssl_cert_nginx
    Function: x509.certificate_managed
        Name: /etc/pki/client1_nginx.crt
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Could not load PEM-encoded public key."
              Could not load PEM-encoded public key.
     Started: 07:24:28.713039
    Duration: 30978.023 ms
     Changes:

Setup

• on-prem machine
• container (LXC running on Proxmox)
• onedir packaging

ssl_cert_nginx state:

ssl_cert_nginx:
x509.certificate_managed:
  - name: {{ pki['certs'][cert]['ssl_cert'] }}
  - ca_server: "{{ pki.ca_server }}"
  - signing_policy: default
  - public_key: {{ pki['certs'][cert]['ssl_key'] }}
  {% if pki['certs'][cert]['append_ca'] %}
  - append_certs:
    - "{{ pki.ca_folder }}/{{ pki.ca_server_files.issuing_salt['cert'] }}"
    - "{{ pki.ca_folder }}/{{ pki.ca_server_files.foobar_ca['cert'] }}"
  {% endif %}
  - CN: {{ salt['grains.get']('id') }}

  # SAN
  {% set san = 'IP:::1,IP:127.0.0.1,DNS:localhost' %}
  {% set san = san + ',DNS:' + salt['grains.get']('id') %}

  # Add custom SAN from pillar
  {% set tmp = namespace(san='') %}
  {% if pki['certs'][cert]['san_entries'] is defined %}
    {% if pki['certs'][cert]['san_entries'] is iterable and pki['certs'][cert]['san_entries'] is not string %}
      {% for san_entry in pki['certs'][cert]['san_entries']|sort %}
        {% set tmp.san = tmp.san + ',' + san_entry %}
      {% endfor %}
    {% endif %}
  {% endif %}

  # Add loadbalancers to SAN if set
  {% if pki['certs'][cert]['add_loadbalancer_to_san'] %}
    {% set san = san + ',DNS:' + foobar.loadbalancer %}
    {% for loadbalancer in foobar.loadbalancers|sort %}
      {% set tmp.san = tmp.san + ',DNS:' + loadbalancer %}
    {% endfor %}
  {% endif %}

  {% set san = san + tmp.san %}

  - subjectAltName: {{ san }}
  - days_valid: 3650
  - days_remaining: 180
  - check_cmd:
      - 'openssl verify {{ pki['certs'][cert]['ssl_cert'] }}'
  - retry:
    - attempts: 3
    - until: true
    - interval: 10
    - splay: 10

_/etc/salt/minion.d/x509.conf_

features:
  x509_v2: true

_/etc/salt/minion.d/signing_policies.conf_

x509_signing_policies:
default:
  - minions: '*'
  - signing_private_key: "{{ pki.ca_server_folder }}/{{ pki.ca_server_files.issuing_salt['key'] }}"
  - signing_cert: "{{ pki.ca_server_folder }}/{{ pki.ca_server_files.issuing_salt['cert'] }}"
  - signing_private_key_passphrase: {{ pki.ca_server_files.issuing_salt['password'] }}
  - basicConstraints: "critical CA:false"
  - keyUsage: "critical keyEncipherment,digitalSignature"
  - extendedKeyUsage: serverAuth,clientAuth
  - subjectKeyIdentifier: hash
  - authorityKeyIdentifier: keyid,issuer:always
  - O: 'FooBar'
  - C: 'US'
  - copypath: /srv/pki/certs_issued/

Steps to Reproduce the behavior
Enable x509_v2, then try to remotely sign certificate via x509.certificate_managed with ca_server set.

Expected behavior
A certificate should be created without issues. I can't find any relevant parts in the "breaking change" that should affect the structure/content of x509.certificate_managed from v1 to v2.

Versions Report

salt-master

Salt Version:
        Salt: 3007.1

Python Version:
      Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions:
        cffi: 1.16.0
    cherrypy: unknown
    dateutil: 2.8.2
   docker-py: Not Installed
       gitdb: Not Installed
   gitpython: Not Installed
      Jinja2: 3.1.4
     libgit2: Not Installed
looseversion: 1.3.0
    M2Crypto: 0.40.1
        Mako: Not Installed
     msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
   packaging: 23.1
   pycparser: 2.21
    pycrypto: Not Installed
pycryptodome: 3.19.1
      pygit2: Not Installed
python-gnupg: 0.5.2
      PyYAML: 6.0.1
       PyZMQ: 25.1.2
      relenv: 0.16.0
       smmap: Not Installed
     timelib: 0.3.0
     Tornado: 6.3.3
         ZMQ: 4.3.4

Salt Package Information:
Package Type: onedir

System Versions:
        dist: debian 11.10 bullseye
      locale: utf-8
     machine: x86_64
     release: 5.15.108-1-pve
      system: Linux
     version: Debian GNU/Linux 11.10 bullseye

salt-minion on client1

Salt Version:
        Salt: 3007.1

Python Version:
      Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions:
        cffi: 1.16.0
    cherrypy: 18.8.0
    dateutil: 2.8.2
   docker-py: Not Installed
       gitdb: Not Installed
   gitpython: Not Installed
      Jinja2: 3.1.4
     libgit2: Not Installed
looseversion: 1.3.0
    M2Crypto: Not Installed
        Mako: Not Installed
     msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
   packaging: 23.1
   pycparser: 2.21
    pycrypto: Not Installed
pycryptodome: 3.19.1
      pygit2: Not Installed
python-gnupg: 0.5.2
      PyYAML: 6.0.1
       PyZMQ: 25.1.2
      relenv: 0.16.0
       smmap: Not Installed
     timelib: 0.3.0
     Tornado: 6.3.3
         ZMQ: 4.3.4

Salt Package Information:
Package Type: onedir

System Versions:
        dist: debian 11.7 bullseye
      locale: utf-8
     machine: x86_64
     release: 5.15.108-1-pve
      system: Linux
     version: Debian GNU/Linux 11.7 bullseye

salt-minion on PKI/CA

Salt Version:
        Salt: 3007.1

Python Version:
      Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions:
        cffi: 1.16.0
    cherrypy: 18.8.0
    dateutil: 2.8.2
   docker-py: Not Installed
       gitdb: Not Installed
   gitpython: Not Installed
      Jinja2: 3.1.4
     libgit2: Not Installed
looseversion: 1.3.0
    M2Crypto: 0.40.1
        Mako: Not Installed
     msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
   packaging: 23.1
   pycparser: 2.21
    pycrypto: Not Installed
pycryptodome: 3.19.1
      pygit2: Not Installed
python-gnupg: 0.5.2
      PyYAML: 6.0.1
       PyZMQ: 25.1.2
      relenv: 0.16.0
       smmap: Not Installed
     timelib: 0.3.0
     Tornado: 6.3.3
         ZMQ: 4.3.4

Salt Package Information:
Package Type: onedir

System Versions:
        dist: debian 11.7 bullseye
      locale: utf-8
     machine: x86_64
     release: 5.15.108-1-pve
      system: Linux
     version: Debian GNU/Linux 11.7 bullseye

Additional context
All of the existing CA/Intermediate CAs etc has not been changed or re-issued after enabling x509_v2. I could not find any "migration" information, so not sure if it's as easy as "just enable v2" or not.

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!