Merge pull request #184 from mirceaulinic/develop

Release 2020.10.0

Dockerfile

@@ -1,8 +1,8 @@
-FROM python:3.6-slim-stretch
+FROM python:3.7.9-slim-stretch
 
 MAINTAINER [email protected]
 
-ARG SALT_VERSION="2019.2.0"
+ARG SALT_VERSION="2019.2.5"
 
 COPY ./ /var/cache/salt-sproxy/
 COPY ./master /etc/salt/master

docs/opts.rst

@@ -126,12 +126,7 @@ already familiar with a vast majority of them from the `salt
         retrieve all this data, *salt-sproxy* initiates the connection with ALL 
         the devices provided through the Roster interface. That means, not only 
         that resources consumption is expected to increase, but also the
-        execution time would similarlly be higher. Depending on your setup and
-        use case, you may want to consider using ``--cache-grains`` and / or 
-        ``--cache-pillar``. The idea is to firstly run ``--invasive-targeting``
-        together with ``--cache-grains`` and / or ``--cache-pillar``, in order
-        to cache your data, and the subsequent executions through *salt-sproxy* 
-        are going to use that data, device target matching included.
+        execution time would similarlly be higher.
 
 .. option:: --preload-targeting
 
@@ -189,6 +184,12 @@ already familiar with a vast majority of them from the `salt
 
 .. option:: --cache-grains
 
+    .. deprecated:: 2020.10.0
+
+        This argument has been defaulted to ``True``, and replaced with 
+        ``--dont-cache-grains``, having a flipped action (i.e., do not attempt
+        to cache the Grains).
+
     Cache the collected Grains. Beware that this option overwrites the existing
     Grains. This may be helpful when using the ``salt-sproxy`` only, but may 
     lead to unexpected results when running in :ref:`mixed-environments`. That 
@@ -198,8 +199,22 @@ already familiar with a vast majority of them from the `salt
     which is a safe operation in this case (i.e., it won't overwrite the Grains 
     of an existing Minion).
 
+.. option:: --dont-cache-grains
+
+    .. versionadded:: 2020.10.0
+
+    Do not attempt to cache the Grains after execution is complete. This is 
+    generally discouraged, particularly if you want to use the ``-G`` or any 
+    targeting combination that requires Grains.
+
 .. option:: --cache-pillar
 
+    .. deprecated:: 2020.10.0
+
+        This argument has been defaulted to ``True``, and replaced with 
+        ``--dont-cache-pillar``, having a flipped action (i.e., do not attempt
+        to cache the Pillar).
+
     Cache the collected Pillar. Beware that this option overwrites the existing
     Pillar. This may be helpful when using the ``salt-sproxy`` only, but may 
     lead to unexpected results when running in :ref:`mixed-environments`. That 
@@ -209,6 +224,14 @@ already familiar with a vast majority of them from the `salt
     which is a safe operation in this case (i.e., it won't overwrite the cached
     Pillar of an existing Minion).
 
+.. option:: --dont-cache-pillar
+
+    .. versionadded:: 2020.10.0
+
+    Do not attempt to cache the Pillar after execution is complete. This is 
+    generally discouraged, particularly if you want to use the ``-I`` or any 
+    targeting combination that requires Pillar.
+
 .. option:: --no-cached-grains
 
     Do not use the cached Grains (i.e., always collect Grains).
@@ -478,6 +501,41 @@ already familiar with a vast majority of them from the `salt
     This option is going to add the salt-sproxy installation path to your
     existing ``file_roots``.
 
+.. option:: -i, --ignore-host-keys
+
+    .. versionadded:: 2020.10.0
+
+    By default ssh host keys are honored and connections will ask for approval.
+    Use this option to disable ``StrictHostKeyChecking``.
+
+.. option:: --no-host-keys
+
+    .. versionadded:: 2020.10.0
+
+    Fully ignores ssh host keys which by default are honored and connections
+    would ask for approval. Useful if the host key of a remote server has
+    changed and would still error with ``--ignore-host-keys``.
+
+.. option:: --identities-only
+
+    .. versionadded:: 2020.10.0
+
+    Execute SSH with ``-o IdentitiesOnly=yes``. This option is intended for
+    situations where ssh-agent offers many different identities and allow ssh
+    to ignore those identities and use the only one specified in options.
+
+.. option:: --priv
+
+    .. versionadded:: 2020.10.0
+
+    Specify the SSH private key file to be used for authentication.
+
+.. option:: --priv-passwd
+
+    .. versionadded:: 2020.10.0
+
+    Specify the SSH private key file's passphrase when required.
+
 .. _logging-opts:
 
 Logging Options

docs/releases/2020.10.0.rst

@@ -0,0 +1,72 @@
+.. _release-2020.10.0:
+
+=========
+2020.10.0
+=========
+
+New Features
+------------
+
+Added new arguments for the SSH functionality (which are similar to the Salt 
+SSH options with the same naming):
+
+- ``--ignore-host-keys``: By default ssh host keys are honored and connections
+  will ask for approval. Use this option to disable ``StrictHostKeyChecking``.
+- ``--no-host-keys``: Fully ignores ssh host keys which by default are honored
+  and connections would ask for approval. Useful if the host key of a remote
+  server has changed and would still error with ``--ignore-host-keys``.
+- ``--identities-only``: Execute SSH with ``-o IdentitiesOnly=yes``. This option
+  is intended for situations where ssh-agent offers many different identities
+  and allow ssh to ignore those identities and use the only one specified
+  in options.
+- ``--priv``: Specify the SSH private key file to be used for authentication.
+- ``--priv-password``: Specify the SSH private key file's passphrase when
+  required.
+
+Any of these CLI arguments can also be provided into the (Master) configuration 
+file as global arguments for all the Minions, as well as individually, for 
+every Proxy Minion.
+
+Deprecations
+------------
+
+The ``--cache-grains`` and ``--cache-pillar`` CLI arguments have been 
+deprecated, as any Minion managed through salt-sproxy now caches the Grains and 
+Pillars by default. If you want to prevent this, you can use the new 
+``--dont-cache-grains`` and / or ``--dont-cache-pillar`` options, or simply 
+configure ``cache_grains: false`` and / or ``cache_pillar: false`` into the 
+(Master) configuration.
+
+See :ref:`opts` for more details.
+
+Improvements
+------------
+
+When using the :ref:`salt-sapi` endpoint for the REST API, in order to have 
+access to the ``sproxy`` and ``sproxy_async`` clients over HTTP calls, any 
+configuration option is now passed on to :ref:`runner`.
+
+Bug Fixes
+---------
+
+- `#176 <https://github.com/mirceaulinic/salt-sproxy/issues/176>`__ "*custom 
+  execution modules not working*".
+- `#169 <https://github.com/mirceaulinic/salt-sproxy/issues/169>`__ "*Pillar 
+  targeting broken*" - which was due to an incorrectly handled caching 
+  mechanism. Pillar targeting should now work well (with the caveats documented 
+  in :ref:`targeting`).
+- `#179 <https://github.com/mirceaulinic/salt-sproxy/issues/179>`__ "*using 
+  salt-sproxy as a replacement for salt-ssh*" - which was fixed in `#185 
+  <https://github.com/mirceaulinic/salt-sproxy/pull/185>`__. Thanks `@dmacvicar 
+  <https://github.com/dmacvicar>`__!
+- `#181 <https://github.com/mirceaulinic/salt-sproxy/issues/181>`__ "*napalm 
+  grains not available during template rendering*" - fixed via `#187 
+  <https://github.com/mirceaulinic/salt-sproxy/pull/187>`__.
+
+Questions
+---------
+
+I would like to engage the community to discuss around the following question:
+*should enforce (internally) to -t 0 (i.e., wait till the device replies) on state.\* functions?*
+`#182 <https://github.com/mirceaulinic/salt-sproxy/issues/182>`__. Any thoughts 
+would be very welcome, or just a simple vote would be sufficient.

docs/releases/index.rst

@@ -10,14 +10,15 @@ Latest Release
 .. toctree::
    :maxdepth: 1
 
-   2020.7.0
+   2020.10.0
 
 Previous Releases
 ^^^^^^^^^^^^^^^^^
 
 .. toctree::
    :maxdepth: 1
 
+   2020.7.0
    2020.3.0
    2020.2.0
    2019.10.0

docs/ssh.rst

@@ -46,6 +46,11 @@ The configuration is aligned to the general Proxy Minion standards: put the
 connection details and credentials under the ``proxy`` key in the Proxy config
 or Pillar.
 
+.. important:
+
+    Local (i.e., per Proxy) option override the global configuration or CLI
+    options.
+
 ``host``
     The IP address or the hostname of the remove machine to manage.
 
@@ -87,6 +92,15 @@ or Pillar.
     situations where ssh-agent offers many different identities and allow ssh
     to ignore those identities and use the only one specified in options.
 
+``ignore_host_keys``: ``False``
+    By default ssh host keys are honored and connections will ask for approval.
+    Use this option to disable ``StrictHostKeyChecking``.
+
+``no_host_keys``: ``False``
+    Fully ignores ssh host keys which by default are honored and connections
+    would ask for approval. Useful if the host key of a remote server has
+    changed and would still error with ``ignore_host_keys``.
+
 ``winrm``: ``False``
     Flag that tells Salt to connect to a Windows machine. This option requires
     the ``saltwinshell`` to be installed.

docs/targeting.rst

@@ -29,17 +29,28 @@ the targeting in *salt-sproxy* comes with some caveats you should be aware of.
 
         salt-sproxy -G netbox:role:router --preview-target
 
+.. _targeting-caveats:
+
 .. seealso::
 
     When targeting making use of Grains or Pillar data that depend on the device 
     characteristics (such as interfaces, IP addresses, OS version, platform 
     details, and so on), or other properties retrieved from other systems, such 
-    as APIs, databases, etc., you may want to look at 
+    as APIs, databases, etc., *salt-sproxy* needs cached data which is obtained 
+    _only_ after the (first) execution is complete. To do so, you have a number 
+    of options: once *salt-sproxy* is installed and configured, you can run 
+    ``salt-sproxy '*' test.ping`` or similar to execute across all the devices 
+    and cache their data.
+
+    Alternatively, you may want to look at
     `--invasive-targeting 
     <https://salt-sproxy.readthedocs.io/en/latest/opts.html#cmdoption-invasive-targeting>`__
     or `--preload-targeting 
     <https://salt-sproxy.readthedocs.io/en/latest/opts.html#cmdoption-preload-targeting>`__
-    options.
+    options. These two options would allow you to work around this caveat, 
+    however bear in mind they'll eveluate - and eventually connect to - every 
+    device *salt-sproxy* is aware of, in order to determine which devices match 
+    your target.
 
 .. _targeting-glob:
 
@@ -105,7 +116,12 @@ This is a tricky subject. Unlike the native Salt, *salt-sproxy* doesn't have
 access to device data before connecting to it (i.e., it can't possibly know 
 device details before even connecting to it). You can however target using 
 Grain data, but there are some caveats, and it's up to you to decide whether 
-you want performance or limit the resource consumption.
+you want performance or limit the resource consumption. Generally, Grain 
+targeting won't work at the first execution, as *salt-sproxy* needs cached 
+data. An alternative would be using the ``--invasive-targeting`` or 
+``--preload-targeting`` options, but that has a price (see 
+https://salt-sproxy.readthedocs.io/en/latest/opts.html#cmdoption-invasive-targeting 
+for more details).
 
 .. seealso::
 
@@ -130,6 +146,11 @@ Grain PCRE
 As the ``grain`` targeting, but instead of exact matching, can match on 
 a regular expression on the Grain value.
 
+.. note::
+
+    This targeting mechanism has the same caveats as the
+    :ref:`targeting-grain`.
+
 Example: match the devices from multiple sites (e.g., ``lon1``, ``lon2``, etc.)
 
 .. code-block:: bash
@@ -138,6 +159,55 @@ Example: match the devices from multiple sites (e.g., ``lon1``, ``lon2``, etc.)
 
 .. _targeting-pillar:
 
+Pillar
+------
+
+Targeting using Pillar data.
+
+Similarly to the Grain targeting, this is possible but with one caveats when
+you're not running active Minions: *salt-sproxy* needs this data cached in 
+order to evaluate the target and determine which devices match, however using 
+the CLI (and configuration file) options ``--invasive-targeting`` or 
+``--preload-targeting``, you can work around this limitation. Once you've 
+executed once, the data will be cached, and you can use it for future targets.
+
+.. hint::
+
+    If you want to target against statically defined Pillar, whenever possible, 
+    static Grains may be a better fit for your use case. Have a look at 
+    :ref:`static-grains`.
+
+Example:
+
+.. code-block:: bash
+
+    salt-sproxy -I proxy:user:salt --preview-target
+
+.. _targeting-pillar-pcre:
+
+Pillar PCRE
+-----------
+
+As the ``pillar`` targeting, but instead of exact matching, can match on 
+a regular expression on the Pillar value.
+
+.. note::
+
+    This targeting mechanism has the same caveats as the
+    :ref:`targeting-pillar`.
+
+.. hint::
+
+    If you want to target against statically defined Pillar, whenever possible, 
+    static Grains may be a better fit for your use case. Have a look at 
+    :ref:`static-grains`.
+
+Example: match the devices from multiple sites, based on the hostname pattern 
+(e.g., ``lon1``, ``lon2``, etc.)
+
+.. code-block:: bash
+
+    salt-sproxy -J proxy:host:.*lon\d --preview-target
 
 .. _targeting-compound:
 

examples/file_roster/README.rst

@@ -120,7 +120,7 @@ of managing the list of devices manually, have it auto-generated:
 
 ``/etc/salt/roster``:
 
-.. code-block:: yaml
+.. code-block:: sls
 
   {%- for i in range(50) %}
   device{{ i }}:

requirements-dev.txt

@@ -1,6 +1,6 @@
 -r requirements.txt
-tox==3.17.0
+tox==3.20.1
 black==19.10b0
-pylint==2.5.3
-SaltPylint==2020.5.22
+pylint==2.6.0
+SaltPylint==2020.9.28
 CherryPy==18.1.0